Live data from Hacker News

ML promises to be profoundly weird

aphyr.com

611–620 of 641 posts

Re: ML promises to be profoundly weird

#611

I so far asked few people to make GPT-5.4 thinking to bullshit (with max 4 pages of prompt), no one can find an example. But the way people speak in general, as well as this post, implies that such a challenge can easily be beaten. If so, I'm not able to find examples.

Here is small example of ChatGPT giving a wrong answer without expressing any doubt (aka bullshitting): https://chatgpt.com/share/69d78ec7-67b0-8395-9fd1-522b760ab5... GPT-5.4 Thinking, Pro account.

Well done, clear, concise and easily verifiable answer to OP's question and ... no response haha.

I'm also just confused by the implication behind their question, is the idea that GPT-5.4 Thinking has never been confidently wrong, ever, about anything?

Re: ML promises to be profoundly weird

#612

Earlier quoted context omitted.

You weren't paying much attention to the "Consider:" part of my previous comment. You don't believe that a well-paid, very careful, high-integrity member of the computer safety community has -on multiple occasions- encountered actual, sustained bullshiting from the latest-available for-pay version of ChatGPT. You don't accept either this fellow's reports or my informed assessment of his computing situation as truthfu…

> On top of that, your goalpost-shifting and general demeanor throughout this conversation simply don't give me the impression that you've much integrity. I'm not spending the equivalent of ten-to-twenty six-packs to reproduce aphyr's work and -given the evidence I have before me- have you reject that, as well. Lol what goal post did I move? I said text only and you rejected it. You can present the example here and l…

> You can't present a _single_ counter example!

Correct. I've presented a _pair_ of examples.

Re: ML promises to be profoundly weird

#613

Earlier quoted context omitted.

>We know that they do not reason because we know the algorithm behind the curtain. In other words, we didn't put the "reasoning algorithm" in LLMs therefore they do not reason. But what is this reasoning algorithm that is a necessary condition for reasoning and how do you know LLMs parameters didn't converge on it in the process of pre-training?

Model parameters are weights, not algorithms. The LLM algorithm is (relatively) fixed: generate the next token according to the existing context, the model weights, and some randomization. That’s it. There is no more algorithm than that. The training parameters can shift the probabilities for predicting a token given the context, but there’s no more to it than that. There is no “reasoning algorithm” in the weights to…

This overly reductive description of LLMs misses the forest for the trees. LLMs are circuit builders, the converged parameters pick out specific paths through the network that define programs. In other words, LLMs are differentiable computers[1]. Analogous to how a CPU is configured by the program state to execute arbitrary programs, the parameters of a converged LLM configure the high level matmul sequences towards a wide range of information dynamics.

Statistics has little relevance to LLM operation. The statistics of the training corpus imparts constraints on the converged circuit dynamics, but otherwise has no representation internally to the LLM.

[1] https://x.com/karpathy/status/1582807367988654081

Re: ML promises to be profoundly weird

#614

I think the discussion has to be more nuanced than this. "LLMs still can't do X so it's an idiot" is a bad line of thought. LLMs with harnesses are clearly capable of engaging with logical problems that only need text. LLMs are not there yet with images, but we are improving with UI and access to tools like figma. LLMs are clearly unable to propose new, creative solutions for problems it has never seen before.

> LLMs with harnesses are clearly capable of engaging with logical problems that only need text.

All of the LLMs are bad at music. They get intervals wrong. They list unsuited songs.

I would not trust them with any domain until proven otherwise.

Re: ML promises to be profoundly weird

#615

Earlier quoted context omitted.

> We are truly in the Information Age now, and I suspect a similar thing will play out for the digital realm. The analogy seems to be backwards though. It would be as if we previously had a scarcity of land and because of that divided it up into private property so markets could maximize crop yield etc. and then someone came up with a way to grow food on asteroids using robots, and that food is only at the 20th perce…

This assumes the limiting factor is content generation, not ability to read and verify. You make the point later in your comment, but consider it a minor issue. “Randos” the actual limits are verification, and then attention. Verification is always more expensive than generation. However, people are happy to consume unverified content which suits their needs. This is why you always needed to subsidize newspapers with…

> This assumes the limiting factor is content generation, not ability to read and verify.

Content generation is the thing copyright applies to. If you want to create a reward system for verification, it's not going to look anything like that.

It mostly looks like things we already have, like laws against pretending you're someone else to trade on their reputation so that people can build a reputation as trustworthy and make money from subscriptions or ads by being the one people to turn to when they want trustworthy information.

> However, people are happy to consume unverified content which suits their needs. This is why you always needed to subsidize newspapers with ads or classifieds.

I suspect the real problem here is the voting thing. When people derive significant value from information they're quite willing to pay for it. Wall St. pays a lot of money for Bloomberg terminals, companies pay to do R&D or market research, individuals often pay for financial software or games and entertainment content etc.

But voting is a collective action problem. Your vote isn't very likely to change the outcome so are you personally going to spend a lot of money to make sure it's informed? For most people the answer is going to be no, so we need something that gives them access to high quality information at minimal cost if we want them to be informed.

Annoyingly one of the common methods of mitigating collective action problems (government funding) has a huge perverse incentive here because the primary thing we want people to be informed about is political issues and official misconduct, so you can't give the incumbent politicians the purse strings for the same reason the First Amendment proscribes them from governing speech.

So you need a way to fund quality reporting the public can access for free. Advertising kind of fit but it never really aligned the incentives. You can often get more views by being entertaining or inflammatory than factual.

The question is basically, who can you get to supply money to fund factual reporting for everyone, whose interest is for it to be accurate rather than biased in favor of the funder's interests? Or, if that's not a thing, whose interests are fairly aligned with those of the general public? Because with that you can use a patronage model, i.e. the content is free to everyone but patrons choose to pay money because they want the work to be done more than they want to not pay.

The obvious answer for "who" is then "the middle class" because they're not so poor they can't pay a few bucks while still consisting of a large diverse group that won't collectively refuse to fund many classes of important reporting. But then we need two things. The first is for the middle class to not get hollowed out, which we're not doing a great job with right now.

And the second is to have a cultural norm where doing this is a thing, i.e. stop teaching people illiterate false dichotomy nonsense where the only two economic camps are "Soviet Communism" in which the government is required to solve everything through central planning and "greed is good" where being altruistic makes you a doofus for not spending all your money on blackjack and cocaine. People rather need to be encouraged to notice that once their basic needs are met, wanting to live in a better world is just as valid a use for free time and disposable income as designer shoes or golf.

Re: ML promises to be profoundly weird

#616

Earlier quoted context omitted.

Crop variety was decreased by the original farming revolution, about 10k years before the industrial revolution. Rather than eating whatever was available, the large majority of the caloric input of an agricultural society comes from a few staple crops optimized for overwinter storability and producing large yields and thus supporting a large number of people. The industrial revolution didn’t qualitatively change far…

This is particularly evident if you had been around rural villages in eastern Europe in the late 00s, particularly those inhabited by elderly people at 70 years old and above. They were still doing subsistence agriculture to supplement their own income well into the 21th century. Of course they didn't grow enough calorie heavy crops like corn, potatoes or wheat to live entirely off the land, but they had enough food…

No, they totally grew enough calories for themselves. My grandparents lived like that. They farmed around 15 hectares, which was actually quite a lot. You can easily grew enough calories for your family on 5 hectares, or even less if you have access to modern cultivars and artificial fertilizer. It’s just even poor people like variety, and will trade some of their crops for stuff they cannot make at home efficiently, like sugar, fish, or candy.

Re: ML promises to be profoundly weird

#617
post #203

Earlier quoted context omitted.

But that is bolted on and is not a core behavior.

Does it matter? Evolution is the brain's very own "pre-training". Hundreds of millions of years of priors hardwired. We can do that for AIs too - pre-train on pure low Kolmogorov complexity synthetics. The AI then "knows things" before it sees any real data. Advantageous sometimes. Hard to pick compute efficient synthetics though.

I think It matters for the question that I was responding to.

Re: ML promises to be profoundly weird

#618

Earlier quoted context omitted.

Model parameters are weights, not algorithms. The LLM algorithm is (relatively) fixed: generate the next token according to the existing context, the model weights, and some randomization. That’s it. There is no more algorithm than that. The training parameters can shift the probabilities for predicting a token given the context, but there’s no more to it than that. There is no “reasoning algorithm” in the weights to…

This overly reductive description of LLMs misses the forest for the trees. LLMs are circuit builders, the converged parameters pick out specific paths through the network that define programs. In other words, LLMs are differentiable computers[1]. Analogous to how a CPU is configured by the program state to execute arbitrary programs, the parameters of a converged LLM configure the high level matmul sequences towards…

> LLMs are circuit builders

I think they are circuit "approximators". In other words, a result of a glorified linear regression..

Re: ML promises to be profoundly weird

#619
post #518

Earlier quoted context omitted.

Compare https://fass.nus.edu.sg/ecs/wp-content/uploads/sites/4/2020/...

First of all, the study is written by an economist, might as well have sent me an Oracle of Delphi pronouncement. And second, he mentions the Malthusian trap being a real thing in his very first sentence, so not sure what I should have gotten out of this.

You could read the whole abstract. Or ask Deep Seek to explain it to you.

Re: ML promises to be profoundly weird

#620
post #33

Earlier quoted context omitted.

It's really the whole tech industry as it exists right now and AI is a victim of bad timing. If this AI had been invented 40 years ago there'd have been a lower ceiling on the damage it could do. Another way of saying that is that capitalism is the real problem, but I was never anti-capitalist in principle, it's just gotten out of hand in the last 5-10 years. (Not that it hadn't been building to that.)

> Another way of saying that is that capitalism is the real problem, but I was never anti-capitalist in principle, it's just gotten out of hand in the last 5-10 years. (Not that it hadn't been building to that.) Capitalism is a tool and it's fine as a tool, to accomplish certain goals while subordinated to other things. Unfortunately it's turned into an ideology (to the point it's worshiped idolatrously by some), and…

Agree. Capitalism is good in limited domains. Applying it generally is ludicrously stupid and will lead to another revolution in the West unless we get it under control
Post reply on HN