Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sigmoids won't save you

astralcodexten.com

Post reply on HN
Print view View original

121–130 of 297 posts

Re: The sigmoids won't save you

#121

Post by kubb »

djeastm wrote: post #106
kubb wrote: post #81

Earlier quoted context omitted.

Gosh it's quite embarassing to have to spell it out, but you inserted the part about Scott's motivations. It can't be found in the text. Neither can any specific discussion of what the dangers are and how we can steer clear. It all comes preplanted in your head. The only thing that Scott is playing on (as far as we can see) is your ingrained fear, by using an ominous headline, and a vague reference to something "scar…

>as far as we can see If only there were a way to see more of Scott's thoughts on the subject of AI..

Sorry, there are many better things to read than Scott's thoughts.

Re: The sigmoids won't save you

#122

Post by noosphr »

This article answers the question in the second paragraph then completely ignores the answer for the rest of it.

>My understanding is that this represents 3-4 “generations” of different technology (propellers, turbojets, etc). Each technology went through normal iterative improvement, then, when it reached its fundamental limits, got replaced by a better technology. The last technology, ramjets, reached its limit at about 3500 km/h, and there wasn’t the economic/regulatory will to develop anything better, so the record stands.

You don't have one sigmoid, you have multiple each stacked on top of each other. Airplanes aren't just one technology they are multiple technologies that happen to do the same thing.

Each one is following a sigmoid perfectly. It only looks exponential(ish) because of unpredictable discoveries that let you switch to another sigmoid that has a higher maximum potential.

The same is true in AI. If you used the same architecture as GPT2 today you're in for a bad time training a new frontier model. It's only because we have dozens of breakthroughs that the capabilities of models have improved as much as they have.

That said exponential and sigmoids are the wrong model to use for growth. Growth is a differential equation. It has independent inputs, it has outputs and some of those outputs are dependent inputs again through causal chains of arbitrary complexity. What happens depends entirely on what the specific DE that governs the given technology is. We can easily have a chaotic system with completely random booms and busts which have no deep fundamental rhyme or reason. We currently call that the economy.

Re: The sigmoids won't save you

#123

Post by Sniffnoy »

paulpauper wrote: post #117
Sniffnoy wrote: post #113

Earlier quoted context omitted.

> FYI: The author has predicted that "AGI" will be here in 1-2 years and has staked his public reputation on it. He is personally invested in trendlines being lindy rather than sigmoid. I mean, that's called "having an opinion".

He wrote articles arguing that pro-AI people are dismissive of risks or even suggesting they are intellectually lazy. He's taken a side. if he's wrong I would hope he owns up to it

> He's taken a side.

Yes, that's called "having an opinion". Typically people writing argumentative pieces are doing so because they have a belief about the matter. I'm not sure what exactly you expect here.

> if he's wrong I would hope he owns up to it

I think Scott Alexander is pretty good about that.

Re: The sigmoids won't save you

#124

Post by sigmoid10 »

dreambuffer wrote: post #99

FYI: The author has predicted that "AGI" will be here in 1-2 years and has staked his public reputation on it. He is personally invested in trendlines being lindy rather than sigmoid. I don't think you can use lindy on trends as if trends are static objects, but that's another conversation.

AGI has become such a meaningless nondescript term, arguing when or how it is here has become pointless. Even OpenAI caved in and removed their AGI clause from their contract with Microsoft because they weren't fully sure that we are not there yet. The original ARC AGI was hailed as proof that AGI is not here yet, but now that ARC 1 and 2 got saturated, noone wanted to consider that perhaps we crossed the point where average humans are getting left behind. Frontier models are primarily limited by context and modality at this point, not by intelligence.

Re: The sigmoids won't save you

#125

Post by Sniffnoy »

noosphr wrote: post #122

This article answers the question in the second paragraph then completely ignores the answer for the rest of it. >My understanding is that this represents 3-4 “generations” of different technology (propellers, turbojets, etc). Each technology went through normal iterative improvement, then, when it reached its fundamental limits, got replaced by a better technology. The last technology, ramjets, reached its limit at…

Yes, I was surprised he never discussed the idea that such exponentials are typically made of stacked sigmoids.

That said... if the exponential is made of stacked sigmoids, it's still an exponential on the whole! The fact that it's made of stacked sigmoids is relevant to the engineers making it, but not so relevant to the users or those otherwise affected by it.

Re: The sigmoids won't save you

#126

Post by LarsDu88 »

Alex_L_Wood wrote: post #103
LarsDu88 wrote: post #83

Earlier quoted context omitted.

I disagree with this. Reinforcement learning with verifiable rewards training is actually the secret sauce that is leading Claude and GPT to automating software engineering tasks. All the easily verifiable domains such as mathematics, coding, and things that can be run inside a reasonable simulation are falling very very fast. By next year if not sooner, mathematicians will be wildly outpaced by LLMs for reasoning.

Coding is anything but “easily” verifiable.

It's extremely verifiable. The reinforcement finetuning strategy I'm referring to involves LLM creating coding tasks with an expected output, implementing the code, and then having a compiler (or interpreter in the case of languages like python) succeed or fail to run the code. Then compare the output to expected output. The verification process (run interpreter + run test) can be done in seconds. One can generate millions of datasets like this for free and there is extensive research showing with the right policy, an agent will be able to learn to reason - first as good as human, and in many cases superior to a human.

Re: The sigmoids won't save you

#127

Post by noosphr »

Sniffnoy wrote: post #125
noosphr wrote: post #122

This article answers the question in the second paragraph then completely ignores the answer for the rest of it. >My understanding is that this represents 3-4 “generations” of different technology (propellers, turbojets, etc). Each technology went through normal iterative improvement, then, when it reached its fundamental limits, got replaced by a better technology. The last technology, ramjets, reached its limit at…

Yes, I was surprised he never discussed the idea that such exponentials are typically made of stacked sigmoids. That said... if the exponential is made of stacked sigmoids, it's still an exponential on the whole! The fact that it's made of stacked sigmoids is relevant to the engineers making it, but not so relevant to the users or those otherwise affected by it.

Only so long as you can keep inventing the next sigmoid in the stack.

Re: The sigmoids won't save you

#128

Post by jfjfnfnttbtg »

tsimionescu wrote: post #100
btilly wrote: post #37

Lindy’s Law is an absolute gem, that I'm keeping. If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th l…

While this is very fun as a mathematical exercise, it's completely irrelevant as a real tool for getting a better understanding of unknown processes in the real world. The law only applies for certain types of processes, and is completely wrong for other types (e.g. a human who has lived 50 years may live 50 more, but one who has lived 100 years will certainly not live 100 more). So the question becomes: what type of…

> The law only applies for certain types of processes

Did you even read the post? It’s an estimate in the context where you have zero information on which to base an accurate estimate. The author’s point is that if you’re making a different estimate you need to actually say what information is informing that.

Human lifespan is obviously not a case where we have zero information, so what is your point in bringing that up?

Re: The sigmoids won't save you

#129

Post by krupan »

inglor_cz wrote: post #18
krupan wrote: post #16

Earlier quoted context omitted.

"There is an international arms race with China" I keep seeing this. Where did it come from? Has China said that they intend to attack other countries using AI? Have other countries declared that they intend to attack China with AI? Also, why does anyone believe that AI could actually be that dangerous, given it's inherent unpredictable and unreliable performance? I would be terrified to rely on AI in a life or death…

It was a metaphor. I meant, and later clarified, an intellectual arms race. BTW your handle is an actual Czech word, minus a diacritic sign ("křupan"), and a bit amusing one. It basically means hillbilly. Not that it matters, just FYI. Anyway: AI will be used in military context, and it probably already is. Both for target acquisition and maybe even driving the weapon itself. As of now, the Ukrainians are almost cert…

That's funny, I was told my real last name is a swear word in Czech

Re: The sigmoids won't save you

#130

Post by Qem »

jsmcgd wrote: post #72

> It’s true that birth rates must eventually flatten out and become sigmoid All positive growth eventually flattens out and becomes sigmoid, but a lot of phenomena experience negative growth and nose dive. No gentle curve, but a hard kink and perfect flat line at zero. Forever. I think it would be a stretch to categorize that pattern as sigmoid. Predicting a sigmoid pattern for negative growth implies some sort of a…

> All positive growth eventually flattens out and becomes sigmoid, but a lot of phenomena experience negative growth and nose dive.

https://en.wikipedia.org/wiki/Seneca_effect

Post reply on HN