Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sigmoids won't save you

astralcodexten.com

Post reply on HN
Print view View original

201–210 of 297 posts

Re: The sigmoids won't save you

#201

Post by riknos314 »

jerf wrote: post #58
btilly wrote: post #37

Lindy’s Law is an absolute gem, that I'm keeping. If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th l…

It's an interesting idea, and it may be something that could be mathematically justified, but I do think this is an abuse of Lindy's Law in the absence of such a justification. Per Wikipedia [1]: "The Lindy effect applies to non-perishable items, like books, those that do not have an "unavoidable expiration date"." And later in the article you can see the mathematical formulation which says the law holds for things w…

> those that do not have an "unavoidable expiration date"."

Try avoiding the heat death of the universe /s

Re: The sigmoids won't save you

#202

Post by mordae »

LarsDu88 wrote: post #84
cyanydeez wrote: post #33

Earlier quoted context omitted.

they already did put a model into the silicon and it's crazy fast. https://chatjimmy.ai/ I'm pretty sure there's a 3 year design goal starting this year that'll do that to any of the qwen, deepseek, etc models. There's a lot you could do with sped up models of these quality. It might even be bad enough that the real bubble is how much we don't need giant data centers when 80-90% of use cases could just be a silicon c…

And this is an asic that is still operating digitally. Imagine a chip with baked it weights that does its math analogue with 20x reduction in number of circuit elements needed to do a multiplication op. If there's a breakthrough in memristors, you could end up with another 20x reduction in circuit elements (get rid of memory bottlnecks, start doing multiplication ops as log transform voltage addition) The ceiling is…

Except weights will be unstable - temperature and frequency dependent - and we still have issues delivering analog circuits reliably to the spec. So it would take multiple attempts.

But yeah, as soon as the digital models start to plateau, ASICs and then this will happen.

Re: The sigmoids won't save you

#203

Post by Smaug123 »

kubb wrote: post #121
djeastm wrote: post #106

Earlier quoted context omitted.

>as far as we can see If only there were a way to see more of Scott's thoughts on the subject of AI..

Sorry, there are many better things to read than Scott's thoughts.

(If you like, you can ask an LLM what he thinks. They're all deeply familiar with his work and can certainly summarise it for you.)

Re: The sigmoids won't save you

#204

Post by simianparrot »

t43562 wrote: post #135
simianparrot wrote: post #64

Earlier quoted context omitted.

Because HN is YCombinator which has invested in probably hundreds of «AI» firms by now. Including OpenAI. Allowing slop articles like this literally prints them evaluation money.

Yes, this is not the place to express skepticism of any kind.

The subtle but consistent downvoting to a score of 0 to -1 feels very botted. No matter how I write the comments, anything counter the "AI" propaganda nets me a 0 or -1 total score. I don't care about the points whatsoever, but I find it interesting how there's never a massive downvote, just enough to keep such comments at 0 to -1.

May be reading into things too much, but it is a bit odd.

Re: The sigmoids won't save you

#205

Post by Smaug123 »

jrflowers wrote: post #112

I like this article about how we should assume, at any given point, that we are exactly halfway through a phenomenon which relies on a single data point on a graph —-that apparently doesn’t need its relevance or importance explained— to illustrate that this is obviously true for AI in particular

I think you may have read a different article from me. The thesis of the article is summarised at the end:

> But if someone claims that the trend toward [X] will never reach some particular scary level, then the burden is on them to explain either:

> If they’re not treating [X] as a black box, and claim to be modeling the dynamics explicitly, then what is their model? Have they calculated the obvious things…

> If they are treating [X] as a black box, why isn’t their default expectation based on Lindy’s Law?

Like, the whole point is that in real life we do actually know things about situations and can model them; we fall back to Lindy's law when we know nothing at all. Further, arguments have justification to deviate from Lindy only when they give specifics about the situation they're modelling.

Re: The sigmoids won't save you

#206

Post by niemandhier »

If you look at problems that can be solved by reasoning in text form or maybe even images, I am more than willing to accept that we simply cannot know when the curve will level.

The situation is drastically different for problems that require interaction was the physical world to determine success.

As soon as you add a powerful simulator for physical problems to the self learning experience of the AI, you are extremely hampered by the large amount of needed computation.

Re: The sigmoids won't save you

#207

Post by mentalgear »

> Why do scaling laws work?

Strictly speaking, the original paradigm of scaling laws doesn't work any more. The assumption that we could achieve better performance simply through "vertical scaling" ie infusing models with exponentially more parameters and pre-training data, is no longer the driving force of AI progress.

Instead, the industry has pivoted toward inference-time scaling. Rather than relying solely on a massive, static neural network, modern architectures allocate more compute during the actual generation process, allowing the model to "think" and verify its logic dynamically.

Furthermore, the latest state-of-the-art models are no longer pure LLMs; they are compound neuro-symbolic systems that integrate external tools like REPLs, databases, and structured skill documentation to archive things pure LLM vertical parameter scaling was not able to do.

Re: The sigmoids won't save you

#208

Post by seanhunter »

BurningFrog wrote: post #175
tsimionescu wrote: post #109

Earlier quoted context omitted.

The argument sounds nice, but it's just wrong. It only works if most processes you're going to encounter that you know nothing about happen to be Lindy processes. If most processes happening around you that you know nothing about are not of that type, then the argument fails.

I understood it as if you know absolutely nothing about a process, your best guess is that it's half done. I don't even think there are any "genuine" Lindy processes. What would those look like? Are they always half done?

> I understood it as if you know absolutely nothing about a process, your best guess is that it's half done.

That is the argument that is being made, but that only holds if the process is drawn from an underlying Pareto distribution with epsilon > 1[1].

As a counterexample, I’m jetlagged and disorientated. I go to sleep and wake up. It’s light outside but I don’t know the time. What’s the best guess of the time of day? By the “Lindy law” the best guess is that the process of daytime is halfway done so if I’m half-way through the day, my best guess is it’s noon.

Clearly that’s not the best guess that could be made. The distribution of times I might wake up is heavily skewed towards the morning, so the best guess is going to be some time in the morning. Now you might argue that we don’t know absolutely nothing about the cycle of the day and night and that’s true. But we also don’t know absolutely nothing about any of the examples in TFA either.

The point is, the times of day I might wake up are not drawn from a pareto distribution with the right parameters so the Lindy Law heuristic completely fails. In TFA the author gives no justification for why the remaining lifespan of the exponential growth of AI might be drawn from such a distribution either, so there’s no reason to think the heuristic will be accurate in that case either.

[1] From https://en.wikipedia.org/wiki/Lindy_effect. epsilon = 1 + 1/p where p is the parameter of the conditional expectation E[T-t|T>t] = p t. So only things with p positive but finite exhibit this effect. If p is negative then the best guess is going to be that the lifetime of the thing will end immediately because we’re already past the expected lifetime, and if p is infinite then the thing will never end so all finite guesses about its length are equally bad. So whether half-way is a good heuristic depends entirely on the underlying process and you’d need to demonstrate that the majority of things have positive p for half-way to be the best guess. That’s far from clear.

Re: The sigmoids won't save you

#210

Post by seanhunter »

XorNot wrote: post #199
dado3212 wrote: post #145

Earlier quoted context omitted.

But if you met an alien who said they'd been alive for 100 years you wouldn't assume they're on the verge of dropping dead: you would assume they live longer. It's a rough rule for when you don't have other information, and if you're arguing against it you need to specify what other information you're using to make that argument.

But it only applies for when you have a single data point: it's more likely to be from the middle of the distribution then the edge. So meeting exactly 1 100 year old alien makes it decent odds that's somewhere near the middle of their lifespan. Because if you grabbed one random human, chances are you'd find someone roughly middle aged.

> Because if you grabbed one random human, chances are you'd find someone roughly middle aged.

That would only be true if the underlying distribution worked that way, which for the human population it doesn’t (global median age is 31, global average expected lifespan is 73), so for humans if you grabbed a random human, chances are you’d find someone less than middle-aged.

Post reply on HN