Could the exclusion of CoT that be because of this recent Anthropic paper? https://assets.anthropic.com/m/71876fabef0f0ed4/original/rea... >We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often…
I have no idea what this means, can someone give the eli5?
Is this the first major flop from Anthropic? This thing is unusable. Slow, awful responses. Since Sonnet 3.5 the only real advance in LLM coding has been Gemini 2.5 Pro's context length. Both complement each other quite well so I'll stick to switch between these 2.
> Slow, awful responses.
Probably there servers cannot handle the traffic today.
An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...
With web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.
For the recent resources it might matter: unless the training data are curated meticulously, they may be "spoiled" by the output of other LLM, or even the previous version of the one that is being trained. That's something what is generally considered dangerous, because it could potentially produce unintentional echo-chamber or even somewhat "incestuously degenerated" new model.
I asked it about Tailwind CSS (since I had problems with Claude not aware of Tailwind 4): > Which version of tailwind css do you know? > I have knowledge of Tailwind CSS up to version 3.4, which was the latest stable version as of my knowledge cutoff in January 2025.
> Which version of tailwind css do you know? LLMs can not reliably tell whether they know or don't know something. If they did, we would not have to deal with hallucinations.
They can if they've been post trained on what they know and don't know. The LLM can first been given questions to test its knowledge and if the model returns a wrong answer, it can be given a new training example with an "I don't know" response.
Me: is 'Claude does not have the ability to run the code it generates yet' a split infinitive? it's clumsy, no? i'd say 'Claude does not yet have the ability to run the code it generates' Claude: You're absolutely right on both counts! "Claude does not have the ability to run the code it generates yet" isn't technically a split infinitive (that would be something like "to quickly run"), but it is definitely clumsy. T…
Still obsequious then. (My opinion is that the original is the superior wording. 'Yet' punctuates the sentence and leaves you considering that it may soon change.)
It still matters for software packages. Particularly python packages that have to do with programming with AI! They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain. It would be great if the models were updating portions of their content more recently than others. For the tailwind example in parent-sibling comment, should absolutely be as up to d…
The fact that things from March are already deprecated is insane.
The Google CoT is so incredibly dumb. I thought my models had been lobotomized until I realized they must be doing some sort of processing on the thing.
it’s fake cot, just like oai
I had assumed it was a way to reduce "hallucinations". Instead of me having to double check every response and prompt it again to clear up the obvious mistakes it just does that in the background with itself for a bit.
Obviously the user still has to double check the response, but less often.
> Which version of tailwind css do you know? LLMs can not reliably tell whether they know or don't know something. If they did, we would not have to deal with hallucinations.
They can if they've been post trained on what they know and don't know. The LLM can first been given questions to test its knowledge and if the model returns a wrong answer, it can be given a new training example with an "I don't know" response.
Oh that's a great idea, just do that for every question the LLM doesn't know the answer to!
That's.. how many questions? Maybe if one model generates all possible questions then