Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Gentle Singularity

blog.samaltman.com

Post reply on HN
Print view View original

341–350 of 557 posts

Re: The Gentle Singularity

#341

Post by myrmidon »

roenxi wrote: post #306
AnthonyMouse wrote: post #97

Earlier quoted context omitted.

The primary operating cost of traditional power plants is fuel, i.e. coal and natural gas. The fuel cost for nuclear power plants is negligible because the energy content is higher by more than a factor of a million. So if you build enough nuclear plants to power the grid, charging per kWh for the electricity is pointless because the marginal cost of the fuel is so low. Meanwhile the construction cost should be on pa…

It is worth noting that the effect is jaw-droppingly stark. The regulators managed to invert the learning curve [0] so the more power plants get built the more expensive it gets! It is one of the most stunning failures of an industrial society in the modern era; the damage this did to us all is huge. It is disheartening that our leadership/society chose to turn their backs on the future and we're all lucky that the C…

> The regulators managed to invert the learning curve [0]

This is conjecture. If you wanted to establish this, you would have to show that cost of (skilled) labor was unchanging or negligible.

It is also important to consider that nuclear power deaths/damages are much more localized and traceable than excess deaths from air pollution, and thus much less acceptable to the voting population-- you could argue that this should not make any difference (I disagree), but I don't want to digress here too much.

> we're all lucky that the Chinese chose a different tack to the West's policy of energy failure

What do you believe that is? Because from my point of view, China generates a negligible amount of electricity from nuclear power (<5%), this is not going to change within the next decades, and the main "purpose" from what I can tell is to in-house reactor/turbine know-how (instead of relying on Alstom/Siemesn).

Re: The Gentle Singularity

#343

Post by paxys »

km3r wrote: post #147
paxys wrote: post #101

Earlier quoted context omitted.

No we'll totally have flying cars and cure cancer and live life in an AR/VR multiverse and make all knowledge 100% free to everyone worldwide. Meanwhile the only real advancements in tech in the last two decades have been smaller computers (smartphones) and ads.

And self driving cars. Real time ray tracing. WiFi. Cloud computing. starlink. Crispr. mRNA vaccines. Blockchain. Voice assistants.

All the significant tech on that list is still "two years away from being two years away".

Re: The Gentle Singularity

#344

Post by quonn »

iNic wrote: post #308
flessner wrote: post #267

> Already we live with incredible digital intelligence, and after some initial shock, most of us are pretty used to it. Very quickly we go from being amazed that AI can generate a beautifully-written paragraph to wondering when it can generate a beautifully-written novel; It was probably around 7 years ago when I first got interested in machine learning. Back then I followed a crude YouTube tutorial which consisted o…

The mere token prediction comment is wrong, but I don't think any of the other comments really explained why. Next token prediction is not what the AI does, but its goal. It's like saying soccer is a boring sport having only ever seen the final scores. The important thing about LLMs is that they can internally represent many different complex ideas efficiently and coherently! This makes them an incredible starting po…

> that they can internally represent many different complex ideas efficiently and coherently

The Transformer circuits[0] suggest that this representation is not coherent at all.

[0] https://transformer-circuits.pub

Re: The Gentle Singularity

#345

Post by FrustratedMonky »

udev4096 wrote: post #325
rcarmo wrote: post #282

This read like a Philip K. Dick, Ubik-style advertisement for a dystopian future, and I’m pretty amazed it is an actual blog post by a corporate leader in 2025. Maybe Sam and Dario should be nominated for Hugos or something…

[flagged]

A crooked CEO doesn't seem to conflict with the concept of "advertisement for a dystopian future".

A CEO can be crooked, and selling a dystopian future.

So, don't think the parent was a delusional post.

Re: The Gentle Singularity

#346

Post by helloplanets »

Sammi wrote: post #329
helloplanets wrote: post #319

Earlier quoted context omitted.

What's your take on Anthropic's 'Tracing the thoughts of a large language model'? [0] > To write the second line, the model had to satisfy two constraints at the same time: the need to rhyme (with "grab it"), and the need to make sense (why did he grab the carrot?). Our guess was that Claude was writing word-by-word without much forethought until the end of the line, where it would make sure to pick a word that rhyme…

What is called "planning" or "thinking" here doesn't seem conceptually much different to me than going from naive breath first search based Dijkstra shortest path search, to adding a heuristics that makes it search in a particular direction first and calling it A*. In both cases you're adding another layer to an existing algorithm in order to make it more effective. Doesn't make either AGI. I'm really no expert in ne…

They definitely do strain the neurology and thinking metaphors in that article. But the Dijkstra's algorithm and A* comparisons are the flipside of that same coin. They aren't trying to make it more effective. And definitely not trying to argue for anything AGI related.

Either way: They're tampering with the inference process, by turning circuits in the LLM on and off, in an attempt to prove that those circuits are related with a specific function. [0]

They noticed that circuits related to a token that is only relevant ~8 tokens forward were already activated on the newline token. Instead of only looking at the sequence of tokens that has been generated so far (aka backwards), and generating the next token based off of that information, the model is activating circuits related to tokens that are not relevant to the next token only, but to specific tokens a handful of tokens after.

So, information related to more than just the next upcoming token (including a reference to just one specific token) is being cached during a newline token. Wouldn't call that thinking, but I don't think calling it planning is misguided. Caching this sort of information in the hidden state would be an emergent feature, rather than a feature that was knowingly aimed at by following a specific training method, unlike with models that do test time compute. (DeepSeek-R1 paper being an example, with a very direct aim at turbocharging test time compute, aka 'reasoning'. [1])

The way they went at defining the function of a circuit, was by using their circuit tracing method, which is open source so you can try it out for yourself. [2] Here's the method in short: [3]

> Our feature visualizations show snippets of samples from public datasets that most strongly activate the feature, as well as examples that activate the feature to varying degrees interpolating between the maximum activation and zero.

> Highlights indicate the strength of the feature’s activation at a given token position. We also show the output tokens that the feature most strongly promotes / inhibits via its direct connections through the unembedding layer (note that this information is typically more meaningful for features in later model layers).

[0]: https://transformer-circuits.pub/2025/attribution-graphs/bio... [1]: https://arxiv.org/pdf/2501.12948 [2]: https://github.com/safety-research/circuit-tracer [3]: https://transformer-circuits.pub/2025/attribution-graphs/met...

Re: The Gentle Singularity

#347

Post by Teever »

sevensor wrote: post #339
Teever wrote: post #270

> There are other self-reinforcing loops at play. The economic value creation has started a flywheel of compounding infrastructure buildout to run these increasingly-powerful AI systems. And robots that can build other robots (and in some sense, datacenters that can build other datacenters) aren’t that far off. > If we have to make the first million humanoid robots the old-fashioned way, but then they can operate the…

Maintaining actual robots in an actual factory is not a job robots are remotely able to do at the moment. They can’t even identify, let alone fix, a problem like a cracked rotor bar in a motor. Not only that, how do they decide to take the machine down to replace a motor or keep running it until the motor fails? And if it fails repeatedly, are the robots going to recognize the need to diagnose the root cause? How? An…

How long do you think it will be before your average consumer has access to a pair of humanoid robots that each cost the price of a luxury vehicle and are able to perform maintenance and replace all the parts of the other?

Re: The Gentle Singularity

#348

Post by jart »

hn_throwaway_99 wrote: post #281
AnthonyMouse wrote: post #97

Earlier quoted context omitted.

The primary operating cost of traditional power plants is fuel, i.e. coal and natural gas. The fuel cost for nuclear power plants is negligible because the energy content is higher by more than a factor of a million. So if you build enough nuclear plants to power the grid, charging per kWh for the electricity is pointless because the marginal cost of the fuel is so low. Meanwhile the construction cost should be on pa…

> Unsurprisingly, this scared the crap out of the fossil fuel industry in the US and countries like Russia that are net exporters of fossil fuels, so they've spent decades lobbying to bind nuclear plant construction up in red tape to prevent them being built and funding anti-nuclear propaganda. This is frankly nonsense, and my hope is that this nonsense is coming from a person too young to remember the real, valid fe…

If you want to see what a thousand square mile uninhabitable wasteland looks like, here's a YouTube video of some guys swimming in a pool underneath the Chernobyl reactor for fun: https://www.youtube.com/watch?v=WOughghZ8To

I'm so tired of hearing about how regulation is this magic salve that saves everything. Regulation is what caused Chernobyl. Soviet regulations mandated that the flawed RBMK reactor design be used. They knew it was flawed and they forced people to use it anyway. Because that's what government does. There's a similar story in western countries, where it hasn't been feasible to use better designs due to antiquated government regulations, it's just no one here has screwed up as badly as the Soviets did.

Re: The Gentle Singularity

#349

Post by ben_w »

While I prefer "event horizon" over "singularity", part of the reason I blogged about this distinction years ago was that the event horizon always seems to be ahead of you as you fall in to a black hole*.

My blog posts didn't age all that well, and I've learned to be a little more sceptical about the speed of technological change, just as the political events over the intervening years have made me more aware of how fast political realities can change: https://benwheatley.github.io/blog/2016/04/12-00.31.55.html and https://benwheatley.github.io/blog/2022/09/20-18.35.10.html

* at least until the rate of change of curvature gets so high you're spaghetti, you're (approximately) co-moving with the light from your own body. This means that when you cross the event horizon, you still see your own legs, even though the space the light is in is moving towards the singularity faster than the light itself moves through that space: https://youtu.be/4rTv9wvvat8?feature=shared&t=516

Re: The Gentle Singularity

#350

Post by ben_w »

sgt101 wrote: post #296
trashtester wrote: post #287

Earlier quoted context omitted.

The "next token prediction" is a distraction. That's not where the interesting part of an AI model happens. If you think of the tokenization near the end as a serializer, something like turning an object model into json, you get a better understanding. The interesting part of a an OOP program is not in the json, but what happens in memory before the json is created. Likewise, the interesting parts of a neural net mod…

>which is in many ways similar to our subconscious thinking this is just made up. - we don't have any useful insight on human subconscious thinking. - we don't have any useful insight on the structures that support human subconscious thinking. - the mechanisms that support human cognition that we do know about are radically different from the mechanisms that current models use. For example we know that biological neu…

The bullet list is a good point, but:

> We also know that biological neural systems continuously learn and adapt, for example in the face of injury. Large models just don't do these things.

This is a deliberate choice on the part of the model makers, because a fixed checkpoint is useful for a product. They could just keep the training mechanism going, but that's like writing code without version control.

> Also this thing about deeper and deeper realities? C'mon, it's surface level association all the way down!

To the extent I agree with this, I think it conflicts with your own point about us not knowing how human minds work. Do I, myself, have deeper truths? Or am myself I making surface level association after surface level association, but have enough levels to make it seem deep? I do not know how many grains make the heap.

Post reply on HN