Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

241–250 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#241

Post by scotty79 »

hackinthebochs wrote: post #197
scotty79 wrote: post #42

Earlier quoted context omitted.

Simple, you just need to turn language into a game. You make models talk to each other, create puzzles for each other's to solve, ask each other to make cases and evaluate how well they were made. Will some of it look like ramblings of pre-scientific philosophers? (or modern ones because philosophy never progressed after science left it in the dust) Sure! But human culture was once there too. And we pulled ourselves…

>And we pulled ourselves out of this nonsense by the bootstraps. Human progress was promoted by having to interact with a physical world that anchored our ramblings and gave us a reward function for coherence and cooperation. LLMs would need some analogous anchoring for it to progress beyond incoherent babble.

True, but LLMs got anchored to reality because we are using them in real world tasks and this connection will only grow richer, wider and faster.

Re: The Bitter Lesson Is Misunderstood

#242

Post by criddell »

bflesch wrote: post #231
criddell wrote: post #220

Earlier quoted context omitted.

> the whole notion of putting one room of your apartment full with random electronics just to cook a meal once in a blue moon is deeply inefficient You don't use your kitchen? After the rooms we sleep in, the kitchen is probably the most used space in my home. We are planning an upcoming renovation of our home and the kitchen is where we plan on spending the most money. > The tableware used should be compatible with…

Yes, of course I use it a lot. It is a great hobby. But only use it because it is kind of forced upon us. It's just so inefficient nowadays. Cooking used to be for the whole homestead or for the large family. Now it is mostly only for the immediate family. All the machines are not utilized properly. When people discussed car sharing it was exactly the same argument and I feel it also applies to kitchens. With the "ta…

I think what a lot of people missed when they were talking about shared cars a few years ago is that people seem to mostly like their cars. They spend far more on them than they need to. The average price for a new car is almost $50k now when a vehicle costing half that would satisfy most people's needs.

I'm guessing people mostly overspend on kitchens as well. When our renovation happens, I'm sure we will and I'll feel pretty good about it.

For cars and kitchens, utilization considerations seem to be ranked way, way below things like comfort and convenience and beauty.

Re: The Bitter Lesson Is Misunderstood

#243

Post by simne »

kushalc wrote: post #49

Hey folks, OOP/original author and 20-year HN lurker here — a friend just told me about this and thought I'd chime in. Reading through the comments, I think there's one key point that might be getting lost: this isn't really about whether scaling is "dead" (it's not), but rather how we continue to scale for language models at the current LM frontier — 4-8h METR tasks. Someone commented below about verifiable rewards…

> if you can find a way to produce verifiable rewards about a target world

I have significant experience on modelling physical world (mostly CFD, but also gamedev - with realistic rigid body collisions and friction).

I admit, exists domain (spectrum of parameters), where CFD and game physics working just well; exists predictable domain (on borders of well working domain), where CFD and game physics working good enough but could show strange things, and exists domain, where you will see lot of bugs.

And, current computing power is so much, that even on small business level (just median gamer desktop), we could save on more than 90% real-world tests with simulations in well working domain (and just avoid use cases in unreliable domains).

So I think, most question is just conservative bosses and investors, who don't believe to engineers and don't understand how to do checks (and tuning) of simulations with real world tests, and what reliable domain is.

Re: The Bitter Lesson Is Misunderstood

#244

Post by coldtea »

giardini wrote: post #222
coldtea wrote: post #142

Earlier quoted context omitted.

> Plato's "Allegory of the cave" was uninteresting and uninformative when I first read it more than 50 years ago. It remains so today. Anything can be uninteresting and uninformative when one doesn't see it's interestingness or can't grok its information. It however stood for millenia as a great device to describe multiple layers of abstractions, deeper reality vs appearance, and so on, with utility as such in countl…

No. the Allegory is a fragment of a poor unfinished story and little more. You don't need it to explain "multiple layers of abstractions, deeper reality vs appearance" as you say. In fact, you don't need it for anything at all except to explain Plato's "Allegory of the cave". Sheesh. coldtea says "...with utility as such in countless domains." So when's the last time you referred to the "Allegory of the cave" in your…

>So when's the last time you referred to the "Allegory of the cave" in your day, other than on HN?

Several times. But it was with broadly educated people, not over-specialized one-dimensional ones.

Re: The Bitter Lesson Is Misunderstood

#245

Post by txrx0000 »

energy123 wrote: post #153
mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

To be fair, GPT-5 didn't start off as a blank slate. The architecture probably encodes a lot, much like how DNA encodes a lot. The former requires human writing to decompress into a human-like thing, the latter requires the Earth environment and a woman to decompress into a human organism.

But it's indeed apples and oranges. There's no good way to estimate the information encoded by the GPT architecture compared to human DNA. We just have to be empirical and look at what the thing can do.

Re: The Bitter Lesson Is Misunderstood

#246

Post by txrx0000 »

nightsd01 wrote: post #92

I am not an expert in AI by any means but I think I know enough about it to comment on one thing: there was an interesting paper not too long ago that showed if you train a randomly-initialized model from scratch on questions, like a bank of physics questions & answers, models will end up with much higher quality if you teach it the simple physics questions first, and then move up to more complex physics questions. T…

This is precisely why chain of thought worked. Written thoughts in plain English is a much higher SNR encoding of the human brain's inner workings than random pages scraped from Amazon. We just want the model to recover the brain, not Amazon's frontend web framework.

Re: The Bitter Lesson Is Misunderstood

#247

Post by jacobolus »

dr_dshiv wrote: post #150
jacobolus wrote: post #107

Earlier quoted context omitted.

And here you can e.g. find Ficino's correspondence translated into English, with commentary, https://archive.org/details/lettersofmarsili0000fici If you make a cursory search you can also find other translations of his works, various biographies, and a wide range of commentary and criticism by later authors. Many of Ficino's originals are also in the corpus of scanned and OCRed or recently republished texts. I'm sure…

Yes, but many of his books are not translated or ocr’d. For instance, La pestilenzia or de mysteriis. And he is one of the most central figures of the renaissance. Less than 20% of neolatin has been digitized, let alone translated. It is fine to question whether including neolatin, Arabic or Sanskrit in AI training will make AI better. But for me, it is a core set of humanism that would be a shame to neglect.

Consiglio contro la pestilenza was apparently written in the Florentine language. You can find a nice scan at https://archive.org/details/ita-bnc-in1-00000486-001/ and the corresponding mediocre OCR at https://archive.org/stream/ita-bnc-in1-00000486-001/ita-bnc-... (using better OCR software could give a version with few errors; I don't think anyone has produced a carefully checked digital text). There's some discussion at https://www.jstor.org/stable/40606241 as well as plenty of other commentary around. If you are trying to figure out specific details about this book, you should just check the book. I wouldn't expect adding a carefully produced copy to an LLM training corpus would make that much difference unless you have niche questions about it.

Re: The Bitter Lesson Is Misunderstood

#249

Post by garspin »

voxic11 wrote: post #203
mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

Maybe human brains are constantly generating (and training on) massive amounts of synthetic data and that is how they get so smart?

I doubt it. Brains run at only a few operations per second.... GPUS at TFLOPS. There just isn't enough bandwidth.

My brain only needs to get mugged in a dark alley by a guy in a hoodie once to learn something.

Re: The Bitter Lesson Is Misunderstood

#250

Post by delusional »

coldtea wrote: post #147
delusional wrote: post #105

Earlier quoted context omitted.

Talking about "time to evolve something" seems patently absurd and unscientific to me. All of nature evolved simultaneously. Nature didn't first make the human body and then go "that's perfect for filling the dishwasher, now to make it talk amongst itself" and then evolve intelligence. It all evolved at the same time, in conjunction. You cannot separate the mind and the body. They are the same physiological and mater…

> Nature didn't first make the human body and then go "that's perfect for filling the dishwasher, now to make it talk amongst itself" and then evolve intelligence. It all evolved at the same time, in conjunction. Nature didn't make decisions about anything. But it also absolutely didn't "all evolved at the same time, in conjunction" (if by that you mean all features, regarding body and intelligence, at the same rate)…

> The substrate is. Doesn't mean the nature of abstract thinking is the same as the nature of the body, in the same way the software as algorithm is not the same as hardware, even if it can only run on hardware.

In this you are positing the existance of a _soul_ that exists separately from the body, and is portable amongst bodies. Analogues to how an algorithm (disembodied software) exists outside of the hardware and is portable amongst it (by embodying it as software).

I don't not agree with that at all, but it's impossible to know of you're right, but I can at least understand why you have a hard time with my argument and the east-west difference if tradition of the existance of a soul is that "obvious" to you.

Post reply on HN