501–510 of 944 posts
Sure as a clock, tick follows tock. Can't imagine trying to build out cost structures, business plans, product launches etc on such rapidly shifting sands. Good that you get more for your money, I suppose. But I get the feeling no model or provider is worth committing to in any serious way.
I just pushed a new release of my LLM CLI tool with support for the new model and the reasoning_effort option: https://llm.datasette.io/en/stable/changelog.html#v0-21 Example usage: llm -m o3-mini 'write a poem about a pirate and a walrus' \ -o reasoning_effort high Output (comparing that with the default reasoning effort): https://github.com/simonw/llm/issues/728#issuecomment-262832... (If anyone has a better demo p…
A reasoning model is not meant for writing poetry. It's not very useful to evaluate it on such tasks.
62% $186 o1 high
60% $18 o3-mini high
57% $5 DeepSeek R1
54% $9 o3-mini medium
52% $14 Sonnet
48% $0 DeepSeek V3
Earlier quoted context omitted.
R1 or R1-Distill? They are not the same thing. I think DeepSeek made a mistake releasing them at the same time and calling them all R1. Full R1 solves this prompt easily for me.
I used R1 hosted at NVidia here: https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard
Earlier quoted context omitted.
By "reasoning" I meant the fact that o*(-mini) does "chain-of-thought", in other words, it prompts itself to "reason" before responding to you, whereas GPT-4o(-mini) just directly responds to your prompt. Thus, it is not appropriate to compare o*(-mini) and GPT-4o(-mini) unless you implement "chain-of-thought" for GPT-4o(-mini) and compare that with o*(-mini). See also: https://docs.anthropic.com/en/docs/build-with-c…
That's like saying you can't compare a sedan to a truck. Sure you can. Even though one is more appropriate for certain tasks than the other.
I do think it is a good metaphor for how all this shakes out though in time.
Earlier quoted context omitted.
> That moat is trivially circumvented by lighting a larger pile of money and less trivially by lighting the pile more efficently. Google with all its money and smart engineers was not able to build a simple chat application.
What do you mean? Gemini app is available on IOS, Android and on the web (as AI Studio https://aistudio.google.com/ ).
I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes...
I haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. In a task like this thread summarization, I typically need to break the document down and loop through chunks to ensure it actually “reads” everything. I might have had to recurse into individual conversations with some small max-depth and leaf count and run inference on each, and then have some aggregati…
o3 does look very promising with regards to large context analysis. I used the same raw data and ran the same prompt as Simon for GPT-4o, GPT-4o mini and DeepSeek R1 and compared their output. You can find the analysis below:
https://beta.gitsense.com/?chat=46493969-17b2-4806-a99c-5d93...
The o3-min model was quite thorough. With reasoning models, it looks like dealing with long context might have gotten a lot better.
Edit:
I was curious if I could get R1 to be more thorough and got the following interesting tidbits.
- Depth Variance: R1 analysis provides more technical infrastructure insights, while o3-mini focuses on developer experience
- Geopolitical Focus: Only R1 analysis addresses China-West tensions explicitly
- Philosophical Scope: R1 contains broader industry meta-commentary absent in o3-mini
- Contrarian Views: o3-mini dedicates specific section to minority opinions
- Temporal Aspects: R1 emphasizes future-looking questions, o3-mini focuses on current implementation
You can find the full analysis at
https://beta.gitsense.com/?chat=95741f4f-b11f-4f0b-8239-83c7...
> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
Funny - I had ChatGPT document some stuff for me this week and asked which responses I preferred as well. Didn’t bother reading either of them, just selected one and went on with my day. If it were me I would have set up a “hey do you mind if we give you two results and you can pick your favorite?” prompt to weed out people like me.
As subsequent models have been released, most of which claim to be better at coding, I've switched cursor to it to give them a try.
o1, o1-pro, deepseek-r1, and the now o3-mini. All of these models suffer from the exact same "adhd." As an example, in a NextJS app, if I do a composer prompt like "on page.tsx [15 LOC], using shadcn components wherever possible, update this page to have a better visual hierarchy."
sonnet nails it almost perfectly every time, but suffers from some date cutoff issues like thinking that shadcn-ui@latest is the repo name.
Every single other model, doesn't matter which, does the following: it starts writing (from scratch), radix-ui components. I will interrupt it and say "DO NOT use radix-ui, use shadcn!" -- it will respond with "ok!" then begin writing its own components from scratch, again not using shadcn.
This is still problematic with o3-mini.
I can't believe it's the models. It must be the instruction-set that cursor is giving it behind the scenes, right? No amount of .cursorrules, or other instruction, seems to get cursor "locked in" the way sonnet just seems to be naturally.
It sucks being stuck on the (now ancient) sonnet, but inexplicably, it remains the only viable coding option for me.
Has anyone found a workaround?