Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sigmoids won't save you

astralcodexten.com

Post reply on HN
Print view View original

101–110 of 297 posts

Re: The sigmoids won't save you

#101

Post by throwawayk7h »

dreambuffer wrote: post #99

FYI: The author has predicted that "AGI" will be here in 1-2 years and has staked his public reputation on it. He is personally invested in trendlines being lindy rather than sigmoid. I don't think you can use lindy on trends as if trends are static objects, but that's another conversation.

Mind you, he is only personally invested insofar as he's staked his reputation on it. Throughout his writing, he expresses the same point over and over again: desperately wants AI to slow down, advocates for politics that would slow it down, and most likely nothing would bring him greater peace than to see a sigmoid curve appear.

Re: The sigmoids won't save you

#102

Post by throwawayk7h »

btilly wrote: post #37

Lindy’s Law is an absolute gem, that I'm keeping. If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th l…

Closely related is Laplace's Rule of Succession[1], which basically says that (in lieu of other information), the odds of something happening next time go down the more times in a row that it doesn't happen (and vice versa).

So for example, the longer a time bomb ticks, the less likely it is to go off any time soon. (Assuming the timer isn't visible.) :)

[1] https://en.wikipedia.org/wiki/Rule_of_succession

Re: The sigmoids won't save you

#103

Post by Alex_L_Wood »

LarsDu88 wrote: post #83
throwaway27448 wrote: post #41

Earlier quoted context omitted.

Even at orders of magnitude greater speed, we've still hit diminishing returns for quality of output. We simply haven't found anything like superhuman reasoning ability, just superhuman (potentially) reasoning speed.

I disagree with this. Reinforcement learning with verifiable rewards training is actually the secret sauce that is leading Claude and GPT to automating software engineering tasks. All the easily verifiable domains such as mathematics, coding, and things that can be run inside a reasonable simulation are falling very very fast. By next year if not sooner, mathematicians will be wildly outpaced by LLMs for reasoning.

Coding is anything but “easily” verifiable.

Re: The sigmoids won't save you

#104

Post by throwawayk7h »

kubb wrote: post #27

If the scary AI is so inevitable, why do you feel such an overwhelming need to convince people about that? Surely you can just wait a bit, and they'll see for themselves.

Yeah! And if climate change is so inevitable, why do the people who want to prevent it from happening seem hell-bent on convincing people that climate change is real?

Re: The sigmoids won't save you

#105

Post by andy99 »

AI has scaled well according to convenient measures. It (neural networks) have the property that whatever you define, they can rapidly be trained master it. We’re able to show that various tasks of increasing complication do not require intelligence and can be framed as autoregressive RL problems. I personally don’t think AI is any closer to sentient intelligence than LeNet; it’s almost trivially clear, we know how it works. So we’re measuring something orthogonal, basically how well a universal function approximator can fit to a function we define, given arbitrary computing power, and calling that progress. What will be really interesting is if we’re able to find a way to properly measure what they can’t do and what’s different about real intelligence.

Edit: in particular I don’t agree with

  But if someone claims that the trend toward increasing AI capabilities will never reach some particular scary level...
One has to agree that the benchmark results are getting “scarier”, which is not automatically implied by finding more goals to optimize for

Re: The sigmoids won't save you

#106

Post by djeastm »

kubb wrote: post #81
mitthrowaway2 wrote: post #76

Earlier quoted context omitted.

In your mind, dangers from AI are imaginary and not even projected, therefore, you don't see any reason to warn about them, because you don't think the dangers are real . You don't believe the road is actually closed up ahead, so you don't think it's necessary to post the sign. In Scott's mind, dangers from AI are not a known fact, but are somewhere between highly probable and a near-certainty. In his mind , there ar…

Gosh it's quite embarassing to have to spell it out, but you inserted the part about Scott's motivations. It can't be found in the text. Neither can any specific discussion of what the dangers are and how we can steer clear. It all comes preplanted in your head. The only thing that Scott is playing on (as far as we can see) is your ingrained fear, by using an ominous headline, and a vague reference to something "scar…

>as far as we can see

If only there were a way to see more of Scott's thoughts on the subject of AI..

Re: The sigmoids won't save you

#107

Post by woeirua »

dreambuffer wrote: post #99

FYI: The author has predicted that "AGI" will be here in 1-2 years and has staked his public reputation on it. He is personally invested in trendlines being lindy rather than sigmoid. I don't think you can use lindy on trends as if trends are static objects, but that's another conversation.

Ok, but you can just look at the METR curve. Mythos saturated the 50% time horizon. The 80% is now at 3 hours. The rate of progress is accelerating not slowing down. There’s no indication yet that this is a sigmoid!

Re: The sigmoids won't save you

#108

Post by andrewflnr »

pyrale wrote: post #98

Such a long article to say that neither side has a fucking idea about what will happen next. While we're at it, the "exponentials are actually sigmoïds" meme is not necessarily true. While exponentials are never exponentials, sigmoids are not guaranteed. Overshoot-and-collapse examples also happen in tech, e.g. the dotcom bubble, or the successive AI winters.

It's really not that long, and is quite clear that its main point is about how to reason when you realize no one actually knows what's going on.

Re: The sigmoids won't save you

#109

Post by tsimionescu »

btilly wrote: post #77
jerf wrote: post #58

Earlier quoted context omitted.

It's an interesting idea, and it may be something that could be mathematically justified, but I do think this is an abuse of Lindy's Law in the absence of such a justification. Per Wikipedia [1]: "The Lindy effect applies to non-perishable items, like books, those that do not have an "unavoidable expiration date"." And later in the article you can see the mathematical formulation which says the law holds for things w…

The analysis in the article explains why it applies to any phenomena that we might be able to notice. The argument given is the same as the one that I first ran across, not by that name, in https://www.nature.com/articles/363315a0 . https://en.wikipedia.org/wiki/Doomsday_argument claims that it was a rediscovery of something that was hypothesized a decade article. I hadn't tried to give it a name, or thought to apply…

The argument sounds nice, but it's just wrong. It only works if most processes you're going to encounter that you know nothing about happen to be Lindy processes. If most processes happening around you that you know nothing about are not of that type, then the argument fails.

Re: The sigmoids won't save you

#110

Post by baxtr »

> The moral of the story is that, even though all exponentials eventually become sigmoids, this doesn’t necessarily happen at the exact moment you’re doing your analysis. Sometimes they stay exponential for much longer than that!

All exponentials eventually become sigmoids? Don’t think this can be true without qualifiers.

Post reply on HN