Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

31–40 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#31

Post by dr_dshiv »

geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

Let’s keep in mind that we don’t have most of the renaissance through the early modern period (1400-1800) because it was published in neolatin with older typefaces— and only about 10% is even digitized.

We probably don’t have most of the Arabic corpus either — and barely any Sanskrit. Classical Chinese is probably also lacking — only about 1% of it is translated to English.

Re: The Bitter Lesson Is Misunderstood

#32

Post by kbenson »

geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

I interpreted it as a roundabout way of increasing quality. Take any given subreddit. You have posts and comments, and scores, but what if the data quality isn't very good overall? What if instead of using it as is, you instead had an AI evaluate and reason about all the posts, and classify them itself based on how useful the posts and comments are, how well they work out in practice (if easily simulated), etc? Essentially you're using the AI to provide a moderated and carefully curated set of information about the information that was already present. If you then ingest this information, does that increase the quality of the data? Probably(?), since you're throwing compute and AI reasoning at the problem ahead of time reducing compute and lowering the low quality data by adding additional high quality data.

Re: The Bitter Lesson Is Misunderstood

#33

Post by Quarrelsome »

I really enjoyed reading this article as I found its content extremely insightful, but I fear I must whine for far too long about something entirely minor.

As someone that didn't go to expensive maths club, the way people who did, talk about maths is disgraceful imho. Consider the equasion in this article:

(C ~ 6 N⋅D)

I can look up the symbol for "roughly equals", that was super cool and is a great part of curiousity. But this _implied_ multiplication between the 6 and the N combined with using a fucking diamond symbol (that I already despise given how long it took me to figure the first time I encountered it) is just gross. I figured it was likely that but then I was like: "but why not just 6ND? Maybe there's a reason why N⋅D but 6 N? Does that mean there's a difference between those operations"?

Thankfully I can use gippity these days to get by, but before gippity I had to look up an entire list of maths symbols to find the diamond symbol to work out what it meant. Its why I love code because there's considerably less implicit behaviour once you slap down the formula into code and you can play with the input/output.

I don't think mathsy people realise how exclusionary their communication is, but its so frustrating when I end up fumbling around in slow-mo when the maths kicks in, because "oh the /2 when discussing logarithms in comp sci is _obvious_, so we just don't put it in the equasion" just kills me. Idiot me, staring at the equasion thinking it actually makes sense without knowing the special maths knowledge of implication means that it actually doesn't solve as it reads on the page. Unless of course you went to expensive maths club where they tell you all this.

What drives me nuts is that every time I spend ages finally grokking something, I realise how obvious it is and how non-trivial it is to explain it simply. Comp sci isn't much better to be honest, where we use CQRS instead of "read here, write there". Which results in thousands of newbies trying to parse the unfathomable complexity of "Command Query Responsibility Segregation" and spending as much time staring at its opaqueness as I did the opening sentence of the wikipedia article on logarithms.

Idk what my point is, I just don't understand what's wrong with 6⋅N⋅D or 6*N*D. Do mathmeticians feel ugly if they write something down like that or smth?

Re: The Bitter Lesson Is Misunderstood

#34

Post by hn_acc1 »

frankenstine wrote: post #28

> The path forward: data alchemists (high-variance, 300% lottery ticket) or model architects (20-30% steady gains) No, the paths forward are: better design, training, feeding in more video, audio, and general data from the outside world. The web is just a small part of our experience. What about apps, webcam streams, radio from all over the world in its many forms, OTA TV, interacting with streaming content via remot…

Reminds me a bit of "Person of Interest" (the TV show).

Re: The Bitter Lesson Is Misunderstood

#35

Post by incompatible »

geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

A lot of newspapers seem to be stuck behind paywalls, even when in the public domain.

Re: The Bitter Lesson Is Misunderstood

#36

Post by brazzy »

geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

The point is that current methods are unable to get more than the current state-of-the-art models' degree of intelligence out of training on the totality of human knowledge. Previously, the amount of compute needed to process that much data was a limit, but not anymore.

So now, in order to progress further, we either have to improve the methods, or synthetically generate more training data, or both.

Re: The Bitter Lesson Is Misunderstood

#37

Post by jacobolus »

dr_dshiv wrote: post #31
geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

Let’s keep in mind that we don’t have most of the renaissance through the early modern period (1400-1800) because it was published in neolatin with older typefaces— and only about 10% is even digitized. We probably don’t have most of the Arabic corpus either — and barely any Sanskrit. Classical Chinese is probably also lacking — only about 1% of it is translated to English.

The volume of text in English and digitized from the past few years dwarfs the volume of Latin text from all time. Unless you are wondering about a very niche historical topic there’s more written in English than Latin about basically everything.

Re: The Bitter Lesson Is Misunderstood

#38

Post by marlott »

FloorEgg wrote: post #13
stego-tech wrote: post #9

Earlier quoted context omitted.

This is my current drum I bang on when an uninformed stakeholder tries shoving LLMs blindly down everyone’s throats: it’s the data, stupid . Current data aggregates outside of industries wholly dependent on it (so anyone not in web advertising, GIS, or intelligence) are garbage , riddled with errors and in awful structures that are opaque to LLMs. For your AI strategy to have any chance of success, your data has to b…

Yes, for me both customers and colleagues continually suggested "hey let's just take all these samples of past work and dump it in the magical black box and then replicate what they have been doing". Instead I developed a UX that made it as easy as possible for people to explain what they want to be done , and a system that then goes and does that. Then we compare the system's output to their historical data and ther…

Interesting! Could you give an example with a bit more specific detail here? I take it there's some kind of work output, like a report, in a semi-structured format, and the goal is to automate creation of these. And you would provide a UX that lets them explain what they want the system to create?

Re: The Bitter Lesson Is Misunderstood

#40

Post by socalgal2 »

FloorEgg wrote: post #10
tliltocatl wrote: post #7

Earlier quoted context omitted.

That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.

Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustnes…

> Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips,

In what sense is this true? That sounds suspiciously like cubic meter of dirt is more advanced than an iPhone because there are 6-7 orders of magnitude more atoms in the dirt.

Post reply on HN