Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Microeconomics of Artificial Intelligence (2025)

direct.mit.edu

Post reply on HN
Print view View original

21–30 of 52 posts

Re: The Microeconomics of Artificial Intelligence (2025)

#21

Post by howunfortunate »

mitxela wrote: post #16
howunfortunate wrote: post #13

Earlier quoted context omitted.

I think that's relatively emergent too though! BERT never really did that (at least to my recollection), presumably because its training was never sufficient for it to develop corrective reasoning in a chain of thought.

BERT isn't a next token predictor. It predicts a single token based on the whole surrounding context in both directions.

I mean sure, but BERT can be and often is used as a next token predictor / generator.

I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).

Re: The Microeconomics of Artificial Intelligence (2025)

#22

Post by howunfortunate »

nightski wrote: post #19
howunfortunate wrote: post #15

Earlier quoted context omitted.

That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention. And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.

I was not saying that they are deterministic, rather that the distributions (aka weights) are fixed. A model as deployed today at anthropic/open ai/etc is not learning beyond the context as far as I know. What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base…

Yes, catastrophic forgetting is absolutely one of the problems that needs to be solved to enable something like this.

My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction.

But as an assessment of the current state, I agree with you, LLMs lag humans severely in ability to self-update.

Re: The Microeconomics of Artificial Intelligence (2025)

#24

Post by bbor »

nadetastic wrote: post #8

> Applied statistics is a far more precise descriptor, “but no one wants to use that term, because it’s not as sexy.” This really hit me some time back when I was explaining AI to a friend. After about 10 mins of rambling about LLMs and mentioning the attention paper like I knew what I was talking about, it ended with “oh so it’s just a really advanced auto correct”

LLMs are applied statistics in the exact same way that you are applied statistics.

Re: The Microeconomics of Artificial Intelligence (2025)

#26

Post by slow_typist »

howunfortunate wrote: post #15
nightski wrote: post #14

Earlier quoted context omitted.

It's a little different than that. Your bundle of nerves and meat is not static. It changes over time. To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try…

That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention. And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.

How are they non-deterministic? Ok there is random involved but for watermarking results the random is substituted by pseudo-random, otherwise it won’t work. If you control the randomness, you should be able to reproduce answers to prompts in equal context 100%.

Re: The Microeconomics of Artificial Intelligence (2025)

#28

Post by imtringued »

howunfortunate wrote: post #15
nightski wrote: post #14

Earlier quoted context omitted.

It's a little different than that. Your bundle of nerves and meat is not static. It changes over time. To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try…

That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention. And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.

It's not "not quite true", it's literally true because alternative architectures like RNNs and Mamba fully update their own internal states, whereas transformers only append to the context.

RNNs and Mamaba do not update their weights, but you could hypothetically scale the internal state to be as big as Fable's and GPT 6's parameters.

Re: The Microeconomics of Artificial Intelligence (2025)

#29

Post by imtringued »

howunfortunate wrote: post #22
nightski wrote: post #19

Earlier quoted context omitted.

I was not saying that they are deterministic, rather that the distributions (aka weights) are fixed. A model as deployed today at anthropic/open ai/etc is not learning beyond the context as far as I know. What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base…

Yes, catastrophic forgetting is absolutely one of the problems that needs to be solved to enable something like this. My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction. But as an…

>My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction.

"LLM" is a branded model as a product. Of course it could be anything, as long as it fulfills the product category.

But we live in reality, we can only look at what models are out there and we see that they don't do any of those things and yet we're supposed to act as if these models already do.

Post reply on HN