Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating GPT-5.6 Sol Ultrafast

cerebras.ai

Post reply on HN
Print view View original

211–220 of 295 posts

Re: Accelerating GPT-5.6 Sol Ultrafast

#211

Post by aenis »

zuzululu wrote: post #90
aenis wrote: post #81

Good news for Intel and AMD. Rught now on large scale codebases the bottleneck is both claude/codex inference, as well as time it takes to run tens of thousands of tests. We put those workloads on dedicated epyc 9005 build machines - but it still takes minutes per run. Those who can afford the fast tokens will be in the market for faster CPU that money can buy today.

why intel and amd ? these are cerebras wafers? i know people are joking about the sol ultrafast prices (its unlikely to be accessible for average joes) but this shows scaling wafer cores works for inference boost which makes me very excited, sol ultrafast will be as slow as it will get if that makes sense. at these token speeds , we will see a much deeper economic impact.

On large codebases, literally anything I throw at Fable or Opus takes at least 20 minutes, 10 of those minutes is spent on running test suites. With this way of working, I put way more emphasis on test coverage, integration tests, behavioral tests, in-browser tests -- and I run them on every change, before any commit. Faster CPU solves this, while faster inference solves the other 10 minutes. I can't imagine how cool it would be to work with this ultrafast mode, without those mental-context-purging pauses.

Re: Accelerating GPT-5.6 Sol Ultrafast

#212

Post by aenis »

gpt5 wrote: post #136
_diyar wrote: post #111

Earlier quoted context omitted.

Right now the bottle neck is not the CPU, so people aren‘t spending big $ on them. But with this ultra fast mode, CPU becomes a bigger part of the bottleneck and thus Intel and AMD can charge more $$$.

I noticed it myself. It's not just that the models are faster, but that by becoming more capable, they can take on larger tasks, which require larger compute.

I am settling on a way of working where during the day I spent the time iterating requirements, designs and low level implementation plans for, say, a dozen features, and then I park them for an unattended build during the night. It works well.

Re: Accelerating GPT-5.6 Sol Ultrafast

#213

Post by planckscnst »

arw0n wrote: post #26
wxw wrote: post #6

> Compared with output speeds reported by Artificial Analysis GPT-5.6 Sol on Ultrafast mode runs 11x faster than Fable 5, and 5x faster than Opus 4.8 on Fast mode. Awesome work. I'm personally very excited for faster models/inference. I think speed is underrated to some degree in the current conversation. For a while, I was using Cursor's Composer quite a lot, even over frontier models, just because of how darn fast…

What do you need speed for? That's a genuine question, I feel like the limiting factor already is my creativity, attention span and budget. And I'm not even yet optimizing cost by batching things like review to slow local models over night, or schedule tasks to take full advantage of my subscriptions.

Do you not have many many separate projects happening in parallel because it takes forever for the model to respond, so you give it your feedback and jump to the next one? That context switch is challenging and expensive. Imagine if your feedback was nearly instantly applied and you could just see the result? It would be more like the holodeck metal table scene in Star Trek "Schisms".

Re: Accelerating GPT-5.6 Sol Ultrafast

#214

Post by LASR »

gtree wrote: post #152
aetherspawn wrote: post #130

Earlier quoted context omitted.

I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed be…

You could say the token usage is "load-bearing".

Its token usage has earned its keep.

Re: Accelerating GPT-5.6 Sol Ultrafast

#217

Post by fireant »

johnfn wrote: post #93

This does look pretty incredible, but don't forget that incredible token thoroughput can only necessarily solve certain bottlenecks. If your e2e tests take an hour, they'll still take an hour after Ultracode. If the agent runs a 10 minute typecheck after a change, that will still take 10 minutes. grep over a massive codebase is still just as slow, etc. I say this not to take away from this accomplishment but just to…

I've been measuring waiting for tool calls/waiting for model response in my OMP with Sol 5.6 and usually it's 85%-95% of time spent waiting for model to respond, so 14x speedup in model perf would still be very significant. YMMV but speeding up tests and improving DX is somewhat well understood.

Re: Accelerating GPT-5.6 Sol Ultrafast

#218

Post by orbital-decay »

woadwarrior01 wrote: post #203
aetherspawn wrote: post #130

Earlier quoted context omitted.

I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed be…

Evidently, Claude's tokenizer vocabulary size is ~15k[1]. On one hand, it's quite mind blowing. On the other hand, Anthropic models' token (in)efficiency makes a lot of sense in that light. [1]: https://xcancel.com/magikarp_tokens/status/20878591737488549...

Not just that, they normalize everything into lowercase and use a special character to capitalize words (what about languages with non-trivial normalization/capitalization?) and mark beginning and end of each word, all of that diluting already small vocabulary. That smells like manual tuning of what should be done statistically, I wonder what technical merit they saw in that - I know they mentioned better generalization, but this is pretty counterintuitive.

Re: Accelerating GPT-5.6 Sol Ultrafast

#219

Post by hadlock »

JohnBooty wrote: post #207
TacticalCoder wrote: post #187

Earlier quoted context omitted.

> Never heard of it before, that's fucking insane. They've been acquired by AMD. Those saying the model sucks are completely missing the point: it was a proof-of-concept. The question is: what happens to a model like Anthropic's Fable 5 that does, what, 70 tokens/s (and requires lots of output tokens) when the latest open-weights model is etched on silicon and does 14 000 tokens/s? Shall the better model still have t…

Shall the better model still have the upper hand or will the raw speed compensate? At 14,000 tokens/sec there's just so much ridiculous stuff that might be possible. Let's assume that this POC proves they can take the next step, and can eventually etch a capable ~27B model into silicon. Let's call it Fred. Ralph loops automatically get real real interesting again. 200x the iteration speed. This is such a clear win I…

At 14,000t/s that's effectively a motor cortex for an android, you no longer need to train the robot to walk, it has a general idea for how to walk (baked into the 1b model), and then just corrects based on sensor input, in real time.
Post reply on HN