Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where does next-token prediction leave us?

pop.rdi.sh

Post reply on HN
Print view View original

91–100 of 221 posts

Re: Where does next-token prediction leave us?

#91

Post by Lerc »

overgard wrote: post #29
Lerc wrote: post #21

Calling this a straw man would be attributing an unnecessarily sturdy construction. Yes there are people making stupid claims on all sides. Attributing phrasing like solved or cooked is as if it is some sort of fanatic specific jargon simply ignores the terminology of different groups of people. I don't use cooked myself, but I am not so ignorant of the younger generation that is see it is just another in the long li…

> Specifically on the topic of next token prediction, we are already past that phase. We really aren't past that phase at all. Reasoning models are just next-token prediction trained in a way where it thinks out loud, essentially. (Source: books on how LLMs actually work, and asking ChatGPT directly!) Harnesses and tool use help a little bit, but it doesn't change fundamentally what an LLM is.

Perhaps you should check your books again, reasoning and reinforcement learning are different things.

Some reasoning is trained by reinforecement you could just finetune reasoning, people have had better results than you would expect by brute forcing inserting tokens to periodically say "wait, let me think."

Reinforcement trains things to produce better results, not move towards a specific correct result. There is no future answer to predict.

Re: Where does next-token prediction leave us?

#93

Post by 0x5FC3 »

osigurdson wrote: post #45

The title is a little misleading as it sounds like it might hint at the end state of next token prediction - i.e. will we reach AGI with it and so on. Instead, this is entirely about "means of production" and "class war" stuff. While that may be coming, it isn't clear that this is the most likely outcome, its just the most pessimistic one. The author might as well have included that it will "be the end of the human r…

I apologise. The title was meant to be ironic, but I see how it is misleading, especially to the HN audience.

Re: Where does next-token prediction leave us?

#94

Post by dingdongditchme »

fny wrote: post #69
overgard wrote: post #40

I have a more depressing theory. I think class is part of it, but I'm starting to suspect that a shockingly large number of people lack the critical thinking skills needed to think out the implications of this stuff. I say that because I've met so many people of the managerial class that seem to think it's going to replace the annoying people they have to pay, but somehow they don't think it's going to come for them…

I don't think its entirely class. I've met many people who have convinced themselves they have some special sauce--even in software! And even those who are insulated like doctors and plumbers don't realize they're still at risk from second order effects. What happens when half of their customers are broke? That possibility is not an exaggeration. Even if everyone re-skills over night, people will default on their mor…

So I am expecting the AI bubble to burst (or at least deflate) some time soon. Perhaps this puts me I an specific camp, I am not sure. But this whole "AI will replace X jobs" does not phase me, not because I think AI is useless. On the contrary I am a daily user, but in my mind, people fail to see that the economy is not built around jobs and capital, but wants (or needs) and trades. Even in a world where everything can be done better by a machine than a human, there will always be the "want" for an item that is handcrafted. AI is yet another tool that accelerates us to satisfy more "wants" and that's great. I'm looking at a whole lot of things that will be available in the future (especially software but not limited to) which are not available today, AI generated or not.

Re: Where does next-token prediction leave us?

#95

Post by bambax »

> The promise of capitalism has always been - you will have a spin at the roulette table.

This may be the promise of the American dream, but it's not the promise of "capitalism". Capitalism promises nothing to the individual. Capitalism means putting machines to work, and using as few people as possible, paying them as little as possible, to operate them. In that sense, AI is capitalists' wet dream: all machines and no people.

The comments on this page so far seem to agree that it all will happen like this. I have doubts. What I see mostly is slop. Slop can replace bullshit jobs, but the point of bullshit jobs is not to produce bullshit, it is to employ people. There is no point in having bullshit jobs done by machines. For the non-bullshit jobs (of which, yes, there may be fewer than we think), slop won't cut it.

Re: Where does next-token prediction leave us?

#96

Post by weatherlite »

fny wrote: post #69
overgard wrote: post #40

I have a more depressing theory. I think class is part of it, but I'm starting to suspect that a shockingly large number of people lack the critical thinking skills needed to think out the implications of this stuff. I say that because I've met so many people of the managerial class that seem to think it's going to replace the annoying people they have to pay, but somehow they don't think it's going to come for them…

I don't think its entirely class. I've met many people who have convinced themselves they have some special sauce--even in software! And even those who are insulated like doctors and plumbers don't realize they're still at risk from second order effects. What happens when half of their customers are broke? That possibility is not an exaggeration. Even if everyone re-skills over night, people will default on their mor…

Perhaps the feeling is that in a competitive society (which most of us live in) as long as things aren't collapsing I don't see why a plumber/doctor should care if a bunch of lawyers and software devs will be out of a job. They will become relatively wealthier even if their income is taking a hit. Their purchasing power should rise if most of white collar work is collapsing.

Re: Where does next-token prediction leave us?

#97

Post by dmje »

mmilunic wrote: post #43
overgard wrote: post #40

I have a more depressing theory. I think class is part of it, but I'm starting to suspect that a shockingly large number of people lack the critical thinking skills needed to think out the implications of this stuff. I say that because I've met so many people of the managerial class that seem to think it's going to replace the annoying people they have to pay, but somehow they don't think it's going to come for them…

Perhaps this is a form of Gell-Mann Amnesia (but kinda inverted) where everyone views AI as too inaccurate for their own niche , but perfectly fine for every other field that they know comparably little about.

I think it’s this but also that we all see the value in our own niche because it’s ours, and have more trouble seeing the values in other niches. So it becomes a self perpetuating positive reinforcement thing.

Re: Where does next-token prediction leave us?

#98

Post by kjkjadksj »

rzmmm wrote: post #83
Gigachad wrote: post #76

Earlier quoted context omitted.

Even if you are a plumber and we don’t automate plumbing, you’ll have half the population displaced and switching to plumbing. Everything that isn’t automated will be over saturated.

Or average joe becomes DIY plumber with help of chatGPT and there is a lot less demand for plumbers then.

How does average joe afford the tools and parts they need with no job employing them?

Re: Where does next-token prediction leave us?

#99

Post by ares623 »

birdsongs wrote: post #71
osigurdson wrote: post #45

The title is a little misleading as it sounds like it might hint at the end state of next token prediction - i.e. will we reach AGI with it and so on. Instead, this is entirely about "means of production" and "class war" stuff. While that may be coming, it isn't clear that this is the most likely outcome, its just the most pessimistic one. The author might as well have included that it will "be the end of the human r…

I would argue we're already in a pseudo class war. Most of us in tech are isolated so far, but the middle class is disappearing worldwide, and the wealth gulf is growing. Most propaganda seems to be centered on increasing division along gender / race / sexuality boundaries, so that infighting keeps people from looking up for too long. But I might just be overly radicalised.

The western world is about to find out first hand why millions of people decide to leave their homes every year to try and start over elsewhere. As one of those millions of people, I can only shake my head. Was working in an office and being in countless standups that bad?

Re: Where does next-token prediction leave us?

#100

Post by brabel »

kqr wrote: post #56
beng-nl wrote: post #50

Earlier quoted context omitted.

I’ve been thinking about this, or a variant of it. Hypothetically, I’m scared and sad that AI can replace me (it currently can’t, not literally, but a lot of my skill and expertise, built up over say 30 years, that used to be valuable and rare is now cheap to get from an AI). Let’s try to see the upside. How ‘powerful’ would it make me if, at the cost of my own edge being dulled, can access everyone else’s edge? I am…

As of today, "minor expert" is the wrong way to phrase it. The flagship LLMs today are at best at an initiate or apprentice level in every field.[1] This is not meant as a derisive remark – having something at hand that is initiate level in every field is remarkable and useful. But it's nowhere near expertise anywhere. [1]: https://entropicthoughts.com/stop-using-junior-and-senior

You seriously think current LLM is just at apprentice level in programming? It can write stuff one shot that I’d expected even some experts to struggle to do even with ample more time allowed.
Post reply on HN