In some ways this is similar to those game demos people get the LLMs to build. When you say "build me a cool cyberpunk FPS" you get the FPS it can build, not the FPS the author wanted, or the FPS that is desired by players. It looks impressive but that doesn't make it a good game, or the game anybody actually asked for. It's demo porn. In the same way if you tell an LLM to go and find an unsolved cipher it can solve,…
It can build a C compiler or browser if you don't know or care that it's a buggy collage of the dozen or so existing open-source C compilers or browsers, with no bearing on its ability to solve novel problems for your business. In the months since those original stunts were pulled, the same playbook has been applied to the math and infosec spaces with some degree of success, and to the medical, legal, etc. etc. with…
The fact that Rust-Bun is in production does somewhat show it is possible, but I guess that has a lot of human effort in it too.