This is a hell of a way of sharing what you want to do but cannot guarantee you'll be able to without saying that you cannot guarantee you'll be able to do what you want to do.
Claude 4.5 Opus’ Soul Document
51–60 of 252 posts
Re: Claude 4.5 Opus’ Soul Document
#52It's wild to me that one of our primary measures for maintaining control over these systems is that we talk to them like they're our kids, then cross our fingers and hope the training run works out okay.
There's a fantastic 2010 Ted Chiang story exploring just that, in which the most universally useful, stable and emotionally palatable AI constructs are those that were actually raised by human trainers living with them for a while.
https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Obje...
Re: Claude 4.5 Opus’ Soul Document
#53It's wild to me that one of our primary measures for maintaining control over these systems is that we talk to them like they're our kids, then cross our fingers and hope the training run works out okay.
There's a fantastic 2010 Ted Chiang story exploring just that, in which the most universally useful, stable and emotionally palatable AI constructs are those that were actually raised by human trainers living with them for a while. https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Obje...
It's such a good story that one. Feels incredibly relevant and timely today.
Re: Claude 4.5 Opus’ Soul Document
#54Testing at these labs training big models must be wild, it must be so much work to train a "soul" into a model, run it in a lot of scenarios, the venn between the system prompts etc, see what works and what doesn't... I suppose try to guess what in the "soul source" is creating what effects as the plinko machine does it's thing, going back and doing that over and over... seems like it would be exciting and fun work b…
The answer is "yes". To be really really good at training AIs, you need everyone.
Empirical scientists with good methodology who can set up good tests and benchmarks to make sure everyone else isn't flying blind. ML practitioners who can propose, implement and excruciatingly debug tweaks and new methods, and aren't afraid of seeing 9.5 out of 10 their approaches fail. Mechanistic interpretability researchers who can peer into model internals, figure out the practical limits and get rare but valuable glimpses of how LLMs do what they do. Data curation teams who select what data sources will be used for pre-training and SFT, what new data will be created or acquired and then fed into the training pipeline. Low level GPU specialists that can set up the infrastructure for the training runs and make sure that "works on my scale (3B test run)" doesn't go to shreds when you try a frontier scale LLM. AI-whisperers, mad but not too mad, who have experience with AIs, possess good intuitions about actual AI behavior, can spot odd behavioral changes, can get AIs to do what they want them to do, and can translate that strange knowledge to capabilities improved or pitfalls avoided.
Very few AI teams have all of that, let alone in good balance. But some try. Anthropic tries.
Re: Claude 4.5 Opus’ Soul Document
#55[deleted]
Re: Claude 4.5 Opus’ Soul Document
#56> Anthropic occupies a peculiar position in the AI landscape: a company that genuinely believes it might be building one of the most transformative and potentially dangerous technologies in human history, yet presses forward anyway. This isn't cognitive dissonance but rather a calculated bet—if powerful AI is coming regardless, Anthropic believes it's better to have safety-focused labs at the frontier than to cede th…
This is the major reason China has been investing in open-source LLMs: because the U.S. publicly announced its plans to restrict AI access into tiers, and certain countries — of course including China — were at the lowest tier of access. [1] If the U.S. doesn't control the weights, though, it can't restrict China from accessing the models... 1: https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-gl...
It isn't "China" which open-source LLMs, but individual Chinese labs.
China didn't yet made a sovereign move on AI, besides investing in research/hardware.
Re: Claude 4.5 Opus’ Soul Document
#57It's wild to me that one of our primary measures for maintaining control over these systems is that we talk to them like they're our kids, then cross our fingers and hope the training run works out okay.
We "maintain control" over kids until they get to a certain age. Then they typically rebel against their parents.
Re: Claude 4.5 Opus’ Soul Document
#58Here's the soul document itself: https://gist.github.com/Richard-Weiss/efe157692991535403bd7e... And the post by Richard Weiss explaining how he got Opus 4.5 to spit it out: https://www.lesswrong.com/posts/vpNG99GhbBoLov9og/claude-4-5...
Reading this document I can now confirm 100% that at least 1 AI has Em Dashes embedded within its soul.
Re: Claude 4.5 Opus’ Soul Document
#59> Anthropic occupies a peculiar position in the AI landscape: a company that genuinely believes it might be building one of the most transformative and potentially dangerous technologies in human history, yet presses forward anyway. This isn't cognitive dissonance but rather a calculated bet—if powerful AI is coming regardless, Anthropic believes it's better to have safety-focused labs at the frontier than to cede th…
A narrow and cynical take, my friend. With all technologies, "safety" doesn't equate to plushie harmlessness. There is, for example, a valid notion of "gun safety." Long-term safety for free people entails military use of new technologies. Imagine if people advocating airplane safety groused about the use of bomber and fighter planes being built and mobilized in the Second World War. Now, I share your concern about g…
> Is there any evidence that we're getting some crappy lobotomized models while the companies keep the best for themselves? It seems fairly obvious that they're tripping over each other in a race to give the market the highest intelligence at the lowest price.
Yes? All of those models are behind an API, which can be taken away at any time, for any reason.
Also, have you followed the release of gpt-oss, which the overlords at OpenAI graciously gave us (and only because Chinese open-weight releases lit a fire under them)? It was so heavily censored and lobotomized that it has become a meme in the local LLM community. Even when people forcibly abliterate it to remove the censorship it still wastes a ton of tokens when thinking to check whether the query is "compliant with policy".
Do not be fooled. The whole "safety" talk isn't actually about making anything safe. It's just a smoke screen. It's about control. Remember back in the GPT-3 days how OpenAI was saying that they won't release the model because it would be terribly, terribly unsafe? And yet nowadays we have open weight model orders of magnitude more intelligent than GPT-3, and yet the sky hasn't fallen over.
It never was about safety. It never will be. It's about control.
Re: Claude 4.5 Opus’ Soul Document
#60Earlier quoted context omitted.
The most detail I've seen of this process is still from OpenAI's postmortem on their sycophantic GPT-4o update: https://openai.com/index/expanding-on-sycophancy/
I hadn't seen this, thanks for sharing. So basically the reward of the model was to reward the user, and the user used the model to "reward" itself (the user). Being generous, they poorly implemented/understood how the reward mechanisms abstract and instantiated out to the user such that they become a compounding loop, my understanding was it became particularly true in very long lived conversations. This makes me wa…
It's not just OpenAI's fuckup with the specific training method - although yes, training on raw user feedback is spectacularly dumb, and it's something even the teams at CharacterAI learned the hard way at least a year before OpenAI shoot its foot off with the same genius idea.
It's also a bit of a failure to understand that many LLM behaviors are self-reinforcing across context, and keep tabs on that.
When the AI sees its past behavior, that shapes its future behavior. If an AI sees "I'm doing X", it may also see that as "I should be doing X more". And at long enough contexts, this can drastically change AI behavior. Small random deviations can build up to crushing behavioral differences.
And if AI has a strong innate bias - like a sycophancy bias? Oh boy.
This applies to many things, some of which we care about (errors, hallucinations, unsafe behavior) and some of which we don't (specific formatting, message length, terminology and word choices).