Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

331–340 of 390 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#331

Post by impossiblefork »

kubb wrote: post #129
impossiblefork wrote: post #116

Earlier quoted context omitted.

I don't agree. At the moment companies like NVIDIA take several times what it costs to make a chip. I think the fair split for the technology contribution is more like 50-50, maybe even 30-70 in favour of the manufacturer. With competition we will actually have the fair split, whatever that is, and thus much lower prices. At the moment, to have a big AI firm, or really AI firm at all, you need to be blessed by NVIDIA…

10-90 is the fair split.

One could hope so, or it would be best for me if it were. I don't think I can count on that though. I don't think I can hope for anything better than 30-70 in my lifetime, and I think 10-90 almost requires you to buy the design firm.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#332

Post by bjourne »

dist-epoch wrote: post #330
bjourne wrote: post #195

The article is a bit naive: > However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use speculative decoding. Speculative decoding leads to a ~3-5x reduction in cost per token. When speculative decoding is implemented on Jalapeño, this will enable Jalapeño to serve tokens even more cost effectively. How much speculative decoding improves throughput is…

In the slides on twitter you can see Jalapeno CAN do speculative decoding. In fact they explicitly mention how compute is disaggregated 3 ways now: prefill, predict, decode, and how a huge Jalapeno advantage is that it uses dark sillicon to switch between these without having to move the KV cache which remains local.

Then the article contradicts the slides because it states that OpenAI choose not to disaggregate prefill and decode. Idk you men with "predict"---conventional LLM serving comprises only two phases.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#333

Post by bjourne »

Traster wrote: post #300

This semi-analysis article reads a lot more like an OpenAI press release than a real analysis. And to be honest some of the statements seem like just straight up lies - they initially claim they were invited to benchmark it, and then half way down switch to claiming that OpenAI provided all the numbers. This really kind of sucks, because I want to read actual detailed nuanced and credible analysis of what's happening…

It's very common in the industry today. They let some journos be the first to break the news. In return the journos write a glowing review. A symbiotic relationship.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#334

Post by aurareturn »

One of the major advantages of Nvidia GPUs is that they can do both training and inference.

If you make an inference only chip, you better be damn sure that it's significantly better than Nvidia's GPUs at it.

Otherwise, it's better to buy Nvidia' GPUs because they're more flexible. You can do a big training run, then use them for inference right after.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#336

Post by Razengan »

jacquesm wrote: post #71
nxtfari wrote: post #13

Earlier quoted context omitted.

Agree, I remember when even half precision made its way into C# sometime around 2020 (I didn’t know much about ML then) and I thought, well I guess that’s a worthwhile tradeoff but I can’t imagine going lower. Lo and behold (1-bit Bonsai) how much lower you could go.

Ternary?

Maybe we have to do what quaternions did for complex numbers and jump straight from 2 to 4

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#337

Post by Razengan »

bayindirh wrote: post #132
ehnto wrote: post #123

Earlier quoted context omitted.

Which in turn reminds me of Soundblaster audio cards! I suspect inference chips are closer to the GPU story than the Soundblaster story though. I remember one soundblaster card I bought came with a Lara Croft demo, that exploited the incredible immersion of real time dynamic reverb. Genuinely I think game audio took a few steps back from that heady era, the innovation in audio likely didn't sell as many cards as grap…

EAX was very powerful in its heyday, but it has died because of a thousand cuts. First we had to have the audio processor. Good EAX was available on top of the line cards, and they were not always cheap. Lower end chips got less features. Then we had to have the speaker setup to have the greatest sound, or needed to get a real 5.1 headphones, which were bulky and never provided the same fidelity. Then Microsoft chang…

Fucking Microsoft killed off a lot of cool shit with potential during the 1990s

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#338

Post by CrimsonRain »

MBCook wrote: post #238
kurthr wrote: post #161

Earlier quoted context omitted.

The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below. https://www.eetimes.com/taalas-specializes-to-extremes-for-e... https://www.turingpost.com/p/taalas https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i... Part of the key is that by moving even f…

But that means your different chips all have different sets of weights and are different generations. If none of that is baked into the chip as now then all the chips are running the latest weights every time. Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.

A model is not useless if it is not sota. Price, and speed are also important.

A 6 months old model that can run at 1/10th hardware and much faster too, can be much more capable than a sota model when you don't have unlimited budget.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#339

Post by einpoklum »

acedTrex wrote: post #149
einpoklum wrote: post #122

I hope the LLM wave will leave GPUs behind to go back to pursue more general-purpose computation rather than spending their die area on multiplying 4-bit-number matrices and such things.

Is that not literally the exect opposite of the direction asics for LLM inference is going?

My point is, that if companies develop ASICs for LLM work, then GPUs will stop being the go-to computation device for these workloads, and that will mean, hopefully, that their architectures will stop being warped so as to cater to LLM work.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#340

Post by dist-epoch »

bjourne wrote: post #332
dist-epoch wrote: post #330

Earlier quoted context omitted.

In the slides on twitter you can see Jalapeno CAN do speculative decoding. In fact they explicitly mention how compute is disaggregated 3 ways now: prefill, predict, decode, and how a huge Jalapeno advantage is that it uses dark sillicon to switch between these without having to move the KV cache which remains local.

Then the article contradicts the slides because it states that OpenAI choose not to disaggregate prefill and decode. Idk you men with "predict"---conventional LLM serving comprises only two phases.

sorry, my mistake, I meant draft not predict

> it states that OpenAI choose not to disaggregate prefill and decode

They disaggregate INSIDE the chip, not by having separate machines for the 3 phases. the slides:

https://x.com/beffjezos/status/2092416851737518190

Post reply on HN