Waiting for the ranking on the lmsys chat arena! The only source of truth
Claude 4
801–810 of 1001 posts
Re: Claude 4
#802Do you know when this will be available on Basalt? They didn't communicate on it yet
Re: Claude 4
#803> Users requiring raw chains of thought for advanced prompt engineering can contact sales So it seems like all 3 of the LLM providers are now hiding the CoT - which is a shame, because it helped to see when it was going to go down the wrong track, and allowing to quickly refine the prompt to ensure it didn't. In addition to openAI, Google also just recently started summarizing the CoT, replacing it with an, in my opi…
Could the exclusion of CoT that be because of this recent Anthropic paper?
https://assets.anthropic.com/m/71876fabef0f0ed4/original/rea...
>We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out.
I.e., chain of thought may be a confabulation by the model, too. So perhaps there's somebody at Anthropic who doesn't want to mislead their customers. Perhaps they'll come back once this problem is solved.
Re: Claude 4
#804It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
> And competitors who _didn't_ jump on the AI bandwagon, but instead kept grinding with a strong focus on quality will eat their lunches.
If the banking industry is any clue they'll get bailout from the government to prevent a "systemic collapse". There is a reason "everyone" is doing it especially with these governments. You get to be cool, you don't risk of missing out and if it blows, you let it blow on the tax payer expense. The only real risk for this system is China because they can now out compete the US industries.
Re: Claude 4
#805It feels like these new models are no longer making order of magnitude jumps, but are instead into the long tail of incremental improvements. It seems like we might be close to maxing out what the current iteration of LLMs can accomplish and we're into the diminishing returns phase. If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut i…
Wouldn't that be the best thing possible for our industry? Watching the bandwagoners and "vibe coders" get destroyed and come begging for actual thinking talent would be delicious. I think the bets are equal on whether later LLMs can unfuck current LLM code to the degree that no one needs to be re-hired... but my bet is on your side, that bad code collapses under its own weight. As does bad management in thrall to tr…
That's not how real life works; you are thinking of a movie. Management will never let down of any power they accumulated until the place is completely ransacked. The Soviet Union is a cautionary tale, a relatively modern event and well documented.
Re: Claude 4
#806Earlier quoted context omitted.
> how any of these companies remain sustainable They don't, they have a big bag of money they are burning through, and working to raise more. Anthropic is in a better position cause they don't have the majority of the public using their free-tier. But, AFAICT, none of the big players are profitable, some might get there, but likely through verticals rather than just model access.
Doesn’t this mean that realistically even if “the bubble never pops”, at some point money will run dry? Or do these people just bet on the post money world of AI?
They will likely just charge a lot more money for these services. Eg, the $200+ per months I think could become more of the entry level in 3-5 years. Saying that smaller models are getting very good, so there could be low margin direct model services and expensive verticals IMO.
Re: Claude 4
#807Ooh, VS Code integration for Claude Code sounds nice. I do feel like Claude Code works better than the native Cursor agent mode. Edit: How do you install it? Running `/ide` says "Make sure your IDE has the Claude Code extension", where do you get that?
[deleted]
Re: Claude 4
#808After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of tha…
I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to ha…
Maybe you are not reading what we are writing :) Here is an article of mine https://omarabid.com/gpt3-now
> But for certain use cases (e.g. simple script, written from scratch), it's absolutely fantastic.
I agree with that. I've found it to be very useful for "yarn run xxx" scripts. Can automate lots of tasks that I wouldn't bother with previously because the cost of coding the automation vs. doing them manually was off.
Re: Claude 4
#809Earlier quoted context omitted.
That is a classic riddle and could easily be part of the training data. Maybe if you change the wording of the logic, then use different names, and change language to a less trained on language than english, it would be meaningful to see if it found the answer using logic rather than pattern recognition
Had you paid more attention, you would have realised it's not the classic riddle, but an already tweaked version that makes it impossible to solve, hence why it is interesting.
Both Claude 4 Sonnet and Opus fail this one, even with extended thinking enabled, and even with a follow-up request to double-check their answers:
“What is heavier, 20 pounds of lead or 20 feathers?”
Re: Claude 4
#810It reminds me, where’s deepseek’s new promised world breaker model?
[dead]