The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe...
[dead]
Claude Opus 4.6
461–470 of 1001 posts
Re: Claude Opus 4.6
#462Is there a good technical breakdown of all these benchmarks that get used to market the latest greatest LLMs somewhere? Preferably impartial.
Re: Claude Opus 4.6
#463The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe...
[flagged]
Re: Claude Opus 4.6
#4645.3 codex https://openai.com/index/introducing-gpt-5-3-codex/ crushes with a 77.3% in Terminal Bench. The shortest lived lead in less than 35 minutes. What a time to be alive!
Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?
We know Open AI got caught getting benchmark data and tuning their models to it already. So the answer is a hard no. I imagine over time it gives a general view of the landscape and improvements, but take it with a large grain of salt.
Re: Claude Opus 4.6
#465Earlier quoted context omitted.
Can you be more specific than this? does it vary in time from launch of a model to the next few months, beyond tinkering and optimization?
Yeah, happy to be more specific. No intention of making any technically true but misleading statements. The following are true: - In our API, we don't change model weights or model behavior over time (e.g., by time of day, or weeks/months after release) - Tiny caveats include: there is a bit of non-determinism in batched non-associative math that can vary by batch / hardware, bugs or API downtime can obviously change…
Do you ever replace ChatGPT models with cheaper, distilled, quantized, etc ones to save cost?
Re: Claude Opus 4.6
#466Claude Code release notes: > Version 2.1.32: • Claude Opus 4.6 is now available! • Added research preview agent teams feature for multi-agent collaboration (token-intensive feature, requires setting CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1) • Claude now automatically records and recalls memories as it works • Added "Summarize from here" to the message selector, allowing partial conversation summarization. • Skills defi…
> Claude now automatically records and recalls memories as it works Neat: https://code.claude.com/docs/en/memory I guess it's kind of like Google Antigravity's "Knowledge" artifacts?
I understand everyone's trying to solve this problem but I'm envisioning 1 year down the line when your memory is full of stuff that shouldn't be in there.
Re: Claude Opus 4.6
#467Earlier quoted context omitted.
A place I worked at used it as part of an interview question (it wasn't some pass/fail thing to get it 100% correct, and was partly a jumping off point to a different question). This was in a city where nearly everyone uses bicycles as everyday transportation. It was surprising how many supposedly mechanical-focused people who rode a bike everyday, even rode a bike to the interview, would draw a bike that would not w…
This is why at my company in interviews we ask people to draw a CPU diagram. You'd be surprised how many supposedly-senior computer programmers would draw a processor that would not work.
If I was asked that question in an interview to be a programmer I'd walk out. How many abstraction layers either side of your knowledge domain do you need to be an expert in? Further, being a good technologist of any kind is not about having arcane details at the tip of your frontal lobe, and a company worth working for would know that.
Re: Claude Opus 4.6
#468Earlier quoted context omitted.
I kinda agree. Their model just doesn't feel "daily" enough. I would use it for any "agentic" tasks and for using tools, but definitely not for day to day questions.
Why? I use it for all and love it. That doesn't mean you have to, but I'm curious why you think it's behind in the personal assistant game.
It's hard to say. Maybe it has to do with the way Claude responds or the lack of "thinking" compared to other models. I personally love Claude and it's my only subscription right now, but it just feels weird compared to the others as a personal assistant.
Re: Claude Opus 4.6
#469Earlier quoted context omitted.
A place I worked at used it as part of an interview question (it wasn't some pass/fail thing to get it 100% correct, and was partly a jumping off point to a different question). This was in a city where nearly everyone uses bicycles as everyday transportation. It was surprising how many supposedly mechanical-focused people who rode a bike everyday, even rode a bike to the interview, would draw a bike that would not w…
This is why at my company in interviews we ask people to draw a CPU diagram. You'd be surprised how many supposedly-senior computer programmers would draw a processor that would not work.
That's reasonable in many cases, but I've had situations like this for senior UI and frontend positions, and they: don't ask UI or frontend questions. And ask their pet low level questions. Some even snort that it's softball to ask UI questions or "they use whatever". It's like, yeah no wonder your UI is shit and now you are hiring to clean it up.
Re: Claude Opus 4.6
#470Claude Code release notes: > Version 2.1.32: • Claude Opus 4.6 is now available! • Added research preview agent teams feature for multi-agent collaboration (token-intensive feature, requires setting CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1) • Claude now automatically records and recalls memories as it works • Added "Summarize from here" to the message selector, allowing partial conversation summarization. • Skills defi…
> Claude now automatically records and recalls memories as it works Neat: https://code.claude.com/docs/en/memory I guess it's kind of like Google Antigravity's "Knowledge" artifacts?
I thought it was already doing this?
I asked Claude UI to clear its memory a little while back and hoo boy CC got really stupid for a couple of days