Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: Claude 4

#751

Post by timschmidt »

troad wrote: post #731
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

> I think that people tend to forget what LLMs really are. [...] They do not have a plan, they do not have thoughts of their own. > A LLM is smart enough to [...] I thought this was an interesting juxtaposition. I think we humans just naturally anthropomorphise everything, and even when we know not to, we do anyway. Your analysis is correct, I think. The reason we find this behaviour frightening is because it appears…

> We've distilled humanity to a grainy facsimile of its most mediocre traits, and now find ourselves alarmed and saddened by what has appeared in the mirror.

I think it's important to point out that this seems to be a near universal failing when humans attempt to examine themselves critically as well. Jung called it the shadow: https://en.wikipedia.org/wiki/Shadow_(psychology) "The shadow can be thought of as the blind spot of the psyche."

There lives everything we do but don't openly acknowledge.

Re: Claude 4

#752

Post by evo_9 »

GuB-42 wrote: post #594
briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

Maybe so, but we’re teaching it these kinds of lines of thinking. And whether or not it creates these thoughts independently and creatively on its own, over the long lifetime of the systems we are the ones introducing dangerous data sets that could eventually cause us as a species harm. Again, I understand that fiction is just fiction, but if that’s the model that these are being trained off of intentionally or otherwise, then that is the model that they will pursue in the future.

Re: Claude 4

#753

Post by jdkee »

anorwell wrote: post #677
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. It's quite reasonable to think that LLMs might plan and have thoughts of their own. No one understands consciousness or the emergent behavior of these models to say with much certainty. It is the "Chinese room" fallacy to assume it's not possible. There's a lot of philosophical deb…

https://transformer-circuits.pub/2025/attribution-graphs/bio...

Re: Claude 4

#754

Post by icpmacdo »

dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase."

SWE bench from ~30-40% to ~70-80% this year

Re: Claude 4

#755

Post by piperswe »

icpmacdo wrote: post #754
dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

How much of that is because the models are optimizing specifically for SWE bench?

Re: Claude 4

#756

Post by timschmidt »

evo_9 wrote: post #752
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

Maybe so, but we’re teaching it these kinds of lines of thinking. And whether or not it creates these thoughts independently and creatively on its own, over the long lifetime of the systems we are the ones introducing dangerous data sets that could eventually cause us as a species harm. Again, I understand that fiction is just fiction, but if that’s the model that these are being trained off of intentionally or other…

Every parent encounters this dilemma. In order to ensure your child can protect itself, you have to teach them about all the dangers of the world. Isolating the child from the dangers only serves to make them more vulnerable. It is an irony that defending one's self from the horrifying requires making a representation of it inside ourselves.

Titration of the danger, and controlled exposure within safer contexts seems to be the best solution anyone's found.

Re: Claude 4

#757

Post by andrei_says_ »

anorwell wrote: post #677
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. It's quite reasonable to think that LLMs might plan and have thoughts of their own. No one understands consciousness or the emergent behavior of these models to say with much certainty. It is the "Chinese room" fallacy to assume it's not possible. There's a lot of philosophical deb…

Seeing faces in the clouds in the sky does not mean the skies are now populated by people.

More likely means that our brains are wired to see faces.

Re: Claude 4

#758

Post by icpmacdo »

piperswe wrote: post #755
icpmacdo wrote: post #754

Earlier quoted context omitted.

"It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase." SWE bench from ~30-40% to ~70-80% this year

How much of that is because the models are optimizing specifically for SWE bench?

not that much because its getting better at all benchmarks

Re: Claude 4

#759

Post by agoodusername63 »

dbingham wrote: post #743

It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…

I think theres still lots of room for huge jumps in many metrics. It feels like not too long ago that DeepSeek demonstrated that there was value in essentially recycling (Stealing, depending on your view) existing models into new ones to achieve 80% of what the industry had to offer for a fraction of the operating cost.

Researchers are still experimenting, I haven't given up hope yet that there will be multiple large discoveries that fundamentally change how we develop these LLMs.

I think I agree with the idea that current common strategies are beginning to scrape the bottom of the barrel though. We're starting to slow down a tad.

Re: Claude 4

#760

Post by [deleted] »

briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

[deleted]
Post reply on HN