31–40 of 51 posts
At the risk of straying into forbidden metacommentary: look, I understand (and also feel) the temptation to have a go at Stephen Wolfram every time he posts something wacky, but… he’s just a guy with a fun hobby, isn’t he? Sure it’s all a bit eccentric and self-indulgent, but so are most hobbies, and in contrast to basically every other rich tech man he isn’t actually hurting anyone. Live and let live I reckon.
I actually think criticism is especially important in the case of high-profile public figures like Wolfram. Far more people in the general public know about him and his highly speculative and unproven theories, while at the same time thousands of serious scientists publishing important and influential work go relatively unrecognized.
Earlier quoted context omitted.
[flagged]
Isn't that also the point of your comment only much lazier?
For all I know, maybe he actually is the world-class genius he believes himself to be, but it's difficult to tell underneath all the blatant self-promotion.
Earlier quoted context omitted.
You can't prove a program completed successfully if you can't prove a program even completes at all. I don't know what properties you think people are trying to pragmatically prove about arbitrary programs but "did the program complete successfully" will always be an important, pragmatic property in software development. Especially in the pragmatic world where so much of our software is chaining smaller programs toge…
I think you have slightly misunderstood the halting problem. There is no computer program H that, for every program P, determines whether P halts. This does not imply that for some particular program Q there is no proof that it halts. In fact in the setting where programs do not take input there is always such a proof (just a trace of the program running to completion). And there are many valid proofs of theorems lik…
You cannot find/prevent every case of an infinite loop at design time/compile time if the only way to determine a complex enough control flow converges towards halting is to run the full program in a live environment.
Earlier quoted context omitted.
Isn't that also the point of your comment only much lazier?
Not sure how you get to "I think I am very smart" from "this guy thinks he is very smart, and his writing invariably highlights that belief to the detriment of whatever he's actually trying to communicate"... For all I know, maybe he actually is the world-class genius he believes himself to be, but it's difficult to tell underneath all the blatant self-promotion.
After Quake was released and the principals of Id Software broke up, John Romero pretty much spent the next few years making an ass of himself during the development of Daikatana. When asked about the delayed and beleaguered development process on Slashdot, John Carmack kind of paid him a backhanded comment saying something like "A properly disciplined Romero is a credit to any team." And that's more or less what Wolfram is: properly disciplined, he would be an asset to any cutting-edge physics research team, but as he his he'd rather go down these rabbit holes than suffer to work with people of more ordinary intellect. And it's his, and the world's, loss really.
Earlier quoted context omitted.
You can't prove a program completed successfully if you can't prove a program even completes at all. I don't know what properties you think people are trying to pragmatically prove about arbitrary programs but "did the program complete successfully" will always be an important, pragmatic property in software development. Especially in the pragmatic world where so much of our software is chaining smaller programs toge…
You have a serious misunderstanding of the consequences of the undecidability of the Halting Problem. The Halting Problem says you can not prove the precise halting behavior in every problem. Precise and every are very important qualifiers. If you sacrifice precise and widen it to: "Halt" and "Maybe run forever, but might just take longer than the age of the universe and is thus irrelevant for actual programs we migh…
Generally speaking, yes, humans want to design programs as best they can in the world of safely terminating applications. Humans aren't perfect at it today, given practical real world data such as CVE databases. Trying to hope humans become perfect to safeguard the world against LLMs like the Googler quote way above is the thing I find futile. Humans aren't going to magically get perfect. We need good enough solutions more than we need perfect solutions. The fear is that LLMs can greatly outpace humans in breaking "good enough" than humans can keep up in building "good enough".
Earlier quoted context omitted.
You have a serious misunderstanding of the consequences of the undecidability of the Halting Problem. The Halting Problem says you can not prove the precise halting behavior in every problem. Precise and every are very important qualifiers. If you sacrifice precise and widen it to: "Halt" and "Maybe run forever, but might just take longer than the age of the universe and is thus irrelevant for actual programs we migh…
But again, the context is not about humans trying very hard to stick to problems in "terminating space" when writing software, but about LLMs with no such compunctions/boundaries finding inputs that break "terminating space assumptions". (Whether or not you also believe that LLMs are capable of staying within the lines of "terminating space" when generating code.) Generally speaking, yes, humans want to design progra…
Even ignoring isolation guarantees that could generically prevent unbounded escalation for arbitrary programs, you can just reject anything that is not "easy" to prove. Humans are really dumb, so anything that is not directly proven or easy to prove automatically is almost certainly not correct, so you can just err on the safe side and just reject them if you care about global correctness.
At the risk of straying into forbidden metacommentary: look, I understand (and also feel) the temptation to have a go at Stephen Wolfram every time he posts something wacky, but… he’s just a guy with a fun hobby, isn’t he? Sure it’s all a bit eccentric and self-indulgent, but so are most hobbies, and in contrast to basically every other rich tech man he isn’t actually hurting anyone. Live and let live I reckon.
Earlier quoted context omitted.
An infinite loop doesn’t mean that you can’t prove whatever property you want to prove about your program, if you’ve designed it for that. The limitations coming from undecidability don’t really affect our everyday applications. Unless your application is exactly to determine whether another arbitrary program will halt, that is. But that’s not what the vast majority of software whose bugs we might be concerned about…
You can't prove a program completed successfully if you can't prove a program even completes at all. I don't know what properties you think people are trying to pragmatically prove about arbitrary programs but "did the program complete successfully" will always be an important, pragmatic property in software development. Especially in the pragmatic world where so much of our software is chaining smaller programs toge…
How in the world does the halting problem enters into the picture if my program is bug free or not?
Am I wrong to conclude that the statement by Google's VP of Security Engineering that "we simply must eliminate every software vulnerability on Earth" (before AI Agents find them) is not only stunningly ambitious, but doomed from the start? https://youtu.be/B_7RpP90rUk (at 3:00)
Kurt Godel says that you are correct.