Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

151–160 of 389 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#151

Post by acedTrex »

vntok wrote: post #76
dpe82 wrote: post #50

Earlier quoted context omitted.

NVidia has enormous operating margins, so a competitive solution doesn't have to match or beat NVidia's scale efficiencies; it just has to beat delivered cost. One objective of the project might be simply to provide credible negotiating leverage when dealing with existing suppliers like NVidia. You don't have to deploy at scale for that to work, but you do have to look like you could if pushed hard enough.

> NVidia has enormous operating margins, so a competitive solution doesn't have to match or beat NVidia's scale efficiencies; it just has to beat delivered cost. But then that means you have no actual moat against the behemot, right? Your competitor can move into the market as soon as they want to, at much better cost (so at slightly better price)... and Nvidia certainly can adapt much faster around hard hardware spe…

But they are also PURCHASING from nvidia so any time nvidia lowers their prices they save money.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#152

Post by LogicFailsMe »

tmp10423288442 wrote: post #20
jimmySixDOF wrote: post #11

I love how now you have to consider the possible s** posting motivation behind analysis of a trillion dollar industry being conducted at a world-class level by a bunch of ex Reddit and 4Chan adjacent mods -- it's one of the best stories in AI that SemiAnalysis is not cut from the same cloth as Gartner McKinsey et al

SemiAnalysis’ founder was roommates with Anthropic people, not OpenAI, so he may be slightly (very slightly) more objective here.

Along with Leopold Aschenbrenner so maybe not so much.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#153

Post by redox99 »

andy_ppp wrote: post #138
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

Yes, they could also sell me GPT Sol 5.6 or 5.7 on a chip and I’d probably buy it. It’s a really really useful model for me, I’m not sure how much better for coding I need it to be. For most things I find Sol good enough with a small amount of coaxing around my tastes.

That'd be ungodly expensive.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#154

Post by m101 »

anthonypasq wrote: post #3

Continued hardware improvements really make it hard for me to believe token prices will not continue to plummet.

With the corollary that old hardware valuations will plummet with them.

Although given we have marginal pricing we need to push through to those lower prices in the face of increasing demand, so timing of this is uncertain and the key to the AI financial markets

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#155

Post by mkl »

fl0id wrote: post #136
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

isn't that what they are doing with cerebras?

No, Cerebras holds the weights in SRAM - they are changeable, not baked in.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#156

Post by jeffbee »

jacquesm wrote: post #147
jeffbee wrote: post #128

Earlier quoted context omitted.

Knuth's base- e proposal enters the chat. They were right about everything 50+ years ago, but they didn't have the budget for the right hardware, had to write conference papers and books instead.

I can totally see how ternary would work from a physical implementation perspective but I have a really hard time visualizing anything using base-e, can you explain how such a thing would work in practice?

No it's impossible. But it would be optimal!

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#157

Post by m101 »

mkw5053 wrote: post #150

Warning, this is a long comment! (I’m trying to stick to sourced facts here and not overstate what they mean) I went down a rabbit hole after watching Dylan Patel on Dwarkesh today: https://www.youtube.com/watch?v=aV26V1UvkJw I was initially just surprised by how bullish Dylan is on OpenAI/Anthropic and how bearish he is on China, despite Chinese labs getting closer to US SOTA while offering inference at dramatically…

Not publicly acknowledging how misallocation of capital may be happening today shows he is corrupt. He’s not that dumb to not know it’s a major risk to the whole story, and is certainly financially incentivised to write as he does.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#159

Post by structural »

andy_ppp wrote: post #138
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

Yes, they could also sell me GPT Sol 5.6 or 5.7 on a chip and I’d probably buy it. It’s a really really useful model for me, I’m not sure how much better for coding I need it to be. For most things I find Sol good enough with a small amount of coaxing around my tastes.

Keep in mind that what previous work has done on a single chip with weights baked in was on a 8b parameter model. Sol is likely something in the 5T parameter range, perhaps higher. Serving the whole thing at BF16 is on the order of $3m in hardware just to serve it at all, and closer to $1-1.5m of hardware if it was being served as NVFP4. And power draw starting at high tens to low hundreds of kilowatts.

Let's say a magic set of chips comes along to host this. Maybe it's 2-3x more efficient in size and power. You're still talking a form factor that's a good chunk of a rack, draws tens of kilowatts, and could actually be sold at a similar if not higher price point because the OPEX is so much lower.

It may be useful but it's certainly uneconomic to spend >$1m to self host the model, plus ongoing power and maintenance costs, plus the cost to adapt whatever building you're in to be able to power it.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#160

Post by rustystump »

LogicFailsMe wrote: post #152
tmp10423288442 wrote: post #20

Earlier quoted context omitted.

SemiAnalysis’ founder was roommates with Anthropic people, not OpenAI, so he may be slightly (very slightly) more objective here.

Along with Leopold Aschenbrenner so maybe not so much.

The guy that was part of FTX, fired from openai for alleged theft, got billions in a hedge fund somehow then lost billions. Why are all these people so scummy? It is like voting Trump three times in a row.
Post reply on HN