Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating GPT-5.6 Sol Ultrafast

cerebras.ai

Post reply on HN
Print view View original

201–210 of 295 posts

Re: Accelerating GPT-5.6 Sol Ultrafast

#201

Post by aexer0e »

TranAndrewA wrote: post #189
dannyw wrote: post #169

Earlier quoted context omitted.

We’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company. The amount of workloads we can shift with an advisor model pattern continues to grow. It’s seriously amazing.

Luna came out about a month ago, you're saying that the cost saving from switching to Luna has saved your company $20 000 000+ in 1 months spending on API usage?

Canva generates billions in annual revenue, it's not unbelievable

Re: Accelerating GPT-5.6 Sol Ultrafast

#202

Post by kanzure »

csallen wrote: post #103

People underestimate the importance of speed on quality of thought, because people underestimate just how much quality is a result of simple iteration. When an LLM thinks, it typically just makes one pass. It outputs tokens from top to bottom, beginning to end, and then it's done. But when people think, especially strong thinkers, we typically iterate and revise our thoughts on the fly. We do numerous passes. We stop…

> Review what you just wrote, according to some appropriate list of evaluation criteria that you come up with first

I get better results from these models when I ask for the appropriate list of evaluation criteria with a fresh context. If you pollute the context with its first iteration, then you are likely to get a worse result when you ask it to come up with the criteria with the first version in the context. Context contamination can unintentionally narrow the expertise of the inquiry (even for meat humanoids).

It's unfortunate that Cerebras disabled new sign-ups for their coder plans. GLM-4.7 on Cerebras via OpenRouter used to be absolutely amazing...

I am very eager to see 15,000 tokens/second eventually, like Talaas but for higher intelligence models. I know a few people working on ASICs in this direction including open-source projects. It's all extremely exciting.

Re: Accelerating GPT-5.6 Sol Ultrafast

#203

Post by woadwarrior01 »

aetherspawn wrote: post #130
iamcoder18 wrote: post #13

I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration. > In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a s…

I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed be…

Evidently, Claude's tokenizer vocabulary size is ~15k[1]. On one hand, it's quite mind blowing. On the other hand, Anthropic models' token (in)efficiency makes a lot of sense in that light.

[1]: https://xcancel.com/magikarp_tokens/status/20878591737488549...

Re: Accelerating GPT-5.6 Sol Ultrafast

#204

Post by sashank_1509 »

navorad772 wrote: post #109
sashank_1509 wrote: post #84

I don’t know if this is that useful for coding. In some autonomous world, where no one check the code and the agent can just spend 10X more time checking its work and leading to better results, yes maybe it is useful. But if humans need to check its work, then 10X speed doesn’t really matter I guess.

Working with agents right now is like working on huge legacy C++ codebase (feel free to insert your nightmares). You change 10 lines of code and then wait 20 minutes before it compiles and runs tests (or write prompt and wait 20 minues for code). External distractions and huge waiting times breaks your flow. Working long time in the flow is what makes mythical 10x programmer. IMHO 10x programmers are not superior in…

Strong disagree, there’s nothing better about an agent generating a 1000 lines in 1 second or 1 minute. I can’t read it that quickly anyway.

Re: Accelerating GPT-5.6 Sol Ultrafast

#205

Post by csallen »

energy123 wrote: post #199
csallen wrote: post #103

People underestimate the importance of speed on quality of thought, because people underestimate just how much quality is a result of simple iteration. When an LLM thinks, it typically just makes one pass. It outputs tokens from top to bottom, beginning to end, and then it's done. But when people think, especially strong thinkers, we typically iterate and revise our thoughts on the fly. We do numerous passes. We stop…

That's an argument for cheap massive parallelism, which is what's behind the recent math breakthroughs, not expensive low latency serial compute.

I vaguely recall some of the recent math breakthroughs being less about parallelism and more about serial compute. A lot of, "Keep trying," "Try harder," "Try another way," etc.

Regardless, I think both things are great, horizontal (more approaches) and vertical (deeper approaches). And both are made more practical with faster inference.

Re: Accelerating GPT-5.6 Sol Ultrafast

#206

Post by moomoo11 »

jgalt212 wrote: post #184
dannyw wrote: post #169

Earlier quoted context omitted.

We’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company. The amount of workloads we can shift with an advisor model pattern continues to grow. It’s seriously amazing.

what has a single company accomplished with tens of millions of token spend?

higher valuation

Re: Accelerating GPT-5.6 Sol Ultrafast

#207

Post by JohnBooty »

TacticalCoder wrote: post #187
perching_aix wrote: post #140

Earlier quoted context omitted.

Never heard of it before, that's fucking insane. Apparently they baked the Llama 3.1 8B model weights [0] into silicon (the actual hardware is called Taalas HC1). I guess for the trillion parameter models this would not scale due to cost? Imagine buying GPT 6 in the form of a PCI-E card, pulling these speeds, with up to 120 cct agent sessions. It'd be beyond wild. [0] the weights are also using some cut down small fo…

> Never heard of it before, that's fucking insane. They've been acquired by AMD. Those saying the model sucks are completely missing the point: it was a proof-of-concept. The question is: what happens to a model like Anthropic's Fable 5 that does, what, 70 tokens/s (and requires lots of output tokens) when the latest open-weights model is etched on silicon and does 14 000 tokens/s? Shall the better model still have t…

    Shall the better model still have 
    the upper hand or will the raw speed 
    compensate?
At 14,000 tokens/sec there's just so much ridiculous stuff that might be possible. Let's assume that this POC proves they can take the next step, and can eventually etch a capable ~27B model into silicon. Let's call it Fred.

Ralph loops automatically get real real interesting again. 200x the iteration speed. This is such a clear win I feel like there's hardly anything to talk about. Instead of one stubborn iterating idiot, you could have dozens of idiots competing in parallel, genetic algorithm style.

The other common orchestration pattern I see is "big model for planning, small parallel subagents implementing, big model reviewing" Today it's Sol dispatching a handful of Luna subagents. Tomorrow maybe it's Sol dispatching as many Fred subagents as it could possibly want.

But what patterns have we not even thought about yet in a world where subagents are 200x faster/cheaper?

What if instead of dispatching single Haiku/Luna/etc subagents, we dispatched "teams" of Fred agents? Maybe each team is 8 Freds. Five come up with competing ideas and the other three vote on a winner.

Or what if they were heterogenous teams? One Luna and a bunch of Freds.

What if instead of a two-tier orchestration system (Sol->Luna) it was three-tier or n-tier? (Sol->Luna->Fred->...Fred)

Those ideas overlap a bit, and crazy shit like Gas Town has already explored even wilder ideas I guess. But man, 14000 tk/sec opens up so much stuff.

Re: Accelerating GPT-5.6 Sol Ultrafast

#208

Post by energy123 »

csallen wrote: post #205
energy123 wrote: post #199

Earlier quoted context omitted.

That's an argument for cheap massive parallelism, which is what's behind the recent math breakthroughs, not expensive low latency serial compute.

I vaguely recall some of the recent math breakthroughs being less about parallelism and more about serial compute. A lot of, "Keep trying," "Try harder," "Try another way," etc. Regardless, I think both things are great, horizontal (more approaches) and vertical (deeper approaches). And both are made more practical with faster inference.

Those results were found with 64 subagents, or with GPT 5.6 Sol Pro which uses parallelism under the hood.

Low latency is a big deal but the immediate use cases are somewhat different in the short-term (more serial coding workflows) rather than pure math research which is effectively massive-scale search through a tree of possibilities, which is where you want throughput and low cost per token, rather than high speed per token.

Re: Accelerating GPT-5.6 Sol Ultrafast

#209

Post by Daneel_ »

TacticalCoder wrote: post #188
Daneel_ wrote: post #168

Earlier quoted context omitted.

If anything, writing "7x" is actually just a poor emulation of writing "7×". Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.

But speech evolves. And we all pronounce "ten ex" , not "ten times" . Actual speech usage very clearly pronounce the letter 'x'. So it's 10x. And no need for Unicode codepoints.

I fully agree that speech evolves, but maybe this is just in the US? I've never really heard anyone say it here in Australia, it's usually "times", although admittedly almost no one uses the phrase/format here.

Re: Accelerating GPT-5.6 Sol Ultrafast

#210

Post by JohnBooty »

andai wrote: post #120
iamcoder18 wrote: post #13

I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration. > In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a s…

I'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.) In fact, I'd say it's overqualified for the kind of work I'm doing, because it s…

Sol medium/high planner orchestrating -> Luna xhigh subagents doing implementation

...has been REALLY good for me. Even on xhigh, Luna is crazy cheap.

Subjectively I'd say it's way better than Sonnet at a fraction of the cost. Luna xhigh can do some decently challenging things on its own, but when orchestrated by a model that is actually good like Sol, I am finding it very very nice.

Post reply on HN