Already test Opus 4 and Sonnet 4 in our SQL Generation Benchmark ( https://llm-benchmark.tinybird.live/ ) Opus 4 beat all other models. It's good.
How does Qwen3 do on this benchmark?
Claude 4
951–960 of 1001 posts
Re: Claude 4
#952Earlier quoted context omitted.
Is this something that the models from 4 months ago were not able to do?
For a fair definition of able, yes. Those models had no ability to engage in a search of emails. What’s special about it is that it required no handholding; that is new.
Is this because the models improved, or the tooling around models improved (both visible and not visible to the end user).
My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.
Re: Claude 4
#953This is the first LLM that has been able to answer my logic puzzle on the first try without several minutes of extended reasoning. > A man wants to cross a river, and he has a cabbage, a goat, a wolf and a lion. If he leaves the goat alone with the cabbage, the goat will eat it. If he leaves the wolf with the goat, the wolf will eat it. And if he leaves the lion with either the wolf or the goat, the lion will eat the…
Answer from Claude 4 Sonnet just now (I copy-pasted your prompt):
> This is a variation of the classic river-crossing puzzle, but with an added complexity - the lion that will eat both the wolf and the goat. Let me work through this step by step: The constraints:
> Goat + Cabbage alone = Goat eats cabbage > Wolf + Goat alone = Wolf eats goat > Lion + Wolf alone = Lion eats wolf > Lion + Goat alone = Lion eats goat
> The solution:
> First trip: Man takes the goat across Left behind: Cabbage, Wolf, Lion (safe - wolf won't eat cabbage, lion won't eat wolf if goat isn't there)
> Second trip: Man returns alone, then takes the wolf across Brings the goat back to prevent wolf from eating it Left behind: Cabbage, Goat, Lion
> Third trip: Man takes the lion across, leaves it with the wolf Left behind: Cabbage, Goat (safe - goat won't eat cabbage if separated properly)
> Fourth trip: Man returns alone, takes the cabbage across Left behind: Goat (alone and safe)
> Fifth trip: Man returns alone, takes the goat across Wait, let me reconsider this more carefully. Actually, there's a fundamental problem: if the lion will eat either the wolf OR the goat when left alone with them, and we need to get all items across, we can never safely leave the lion with either animal.
> The puzzle as stated appears to have no solution, because the lion creates an impossible constraint - it cannot be left alone with either the wolf or the goat on either side of the river, making it impossible to transport all items safely.
It got it right, eventually which is impressive. However, it's ability to retain proper context is still a problem (it took the goat on the first trip, then thinks the goat is still on the same side of the river as the other things.
Re: Claude 4
#954Earlier quoted context omitted.
I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to ha…
Maybe you are not reading what we are writing :) Here is an article of mine https://omarabid.com/gpt3-now > But for certain use cases (e.g. simple script, written from scratch), it's absolutely fantastic. I agree with that. I've found it to be very useful for "yarn run xxx" scripts. Can automate lots of tasks that I wouldn't bother with previously because the cost of coding the automation vs. doing them manually was…
That was a fun read - thanks.
Re: Claude 4
#955Earlier quoted context omitted.
I was thinking that too, grok can comment on things that have only just broke out hours earlier, cutoff dates don't seem to matter much
Yeah, it seems pretty up-to-date with Elon's latest White Genocide and Holocaust Denial conspiracy theories, but it's so heavy handed about bringing them up out of the blue and pushing them in the middle of discussions about the Zod 4 and Svelte 5 and Tailwind 4 that I think those topics are coming from its prompts, not its training.
while this is obviously a very damning example, tbf it does seem to be an extreme outlier.
Re: Claude 4
#956Earlier quoted context omitted.
Especially since the EU just made open source contributors liable for cybersecurity (Cyber Resilience Act). Just let AI contribute and ur good
Didn’t they make an exception for open-source projects? https://opensource.org/blog/the-european-regulators-listened...
“Anyone opensourcing anything while in the course of ‘commercial activity’ will be fully liable. Effectively they rugpulled the Apache2 / MIT licenses... all opensource released by small businesses is fucked. where the was no red tape now there is infinite liability”
This is my current understanding, from a friend not a lawyer. Would appreciate any insight from folks here.
Re: Claude 4
#957Earlier quoted context omitted.
One thing I'm 100% is that a cut off date doesn't exist for any large model, or rather there is no single date since it's practically almost impossible to achieve that.
Indeed. It’s not possible stop the world and snapshot the entire internet in a single day. Or is it?
You can trivially maximal bound it, though. If the training finished today, then today is a cutoff date.
Re: Claude 4
#958Earlier quoted context omitted.
Could the exclusion of CoT that be because of this recent Anthropic paper? https://assets.anthropic.com/m/71876fabef0f0ed4/original/rea... >We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often…
I have no idea what this means, can someone give the eli5?
I don't either, but chain of thought is obviously bullshit and just more LLM hallucination.
LLMs will routinely "reason" through a solution and then proceed to give out a final answer that is completely unrelated to the preceding "reasoning".
Re: Claude 4
#959An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025) https://docs.anthropic.com/en/docs/about-claude/models/overv...
One thing I'm 100% is that a cut off date doesn't exist for any large model, or rather there is no single date since it's practically almost impossible to achieve that.
But I think the general meaning of a cutoff date, D, is:
The model includes nothing AFTER date D
and not
The model includes everything ON OR BEFORE date D
Right? Definitionally, the model can't include anything that happened after training stopped.
Re: Claude 4
#960Earlier quoted context omitted.
One thing I'm 100% is that a cut off date doesn't exist for any large model, or rather there is no single date since it's practically almost impossible to achieve that.
Indeed. It’s not possible stop the world and snapshot the entire internet in a single day. Or is it?
That's... not what a cutoff date means. Cutoff date is an upper bound, not a promise that the model is trained on every piece of information set in a fixed form before that date.