Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

361–370 of 390 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#361

Post by MBCook »

kurthr wrote: post #277
MBCook wrote: post #238

Earlier quoted context omitted.

But that means your different chips all have different sets of weights and are different generations. If none of that is baked into the chip as now then all the chips are running the latest weights every time. Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.

Since the current NVL72 are still at the ~15%/yr failure rate it's not clear your new data center is going to have half it's compute in 3 years. If you're still running H100s they draw >10x kWh/Mtoken as new designs. All of these systems become dated, but not all of them require entirely new infrastructure. If a ROM rack running a near frontier agent model at >10ktoken/sec costs What these don't do is TRAINING, they…

They fail that often? Damn.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#362

Post by thesz »

bmulholland wrote: post #97
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this. Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a mont…

I made some analysis half a year ago: https://news.ycombinator.com/item?id=47109252

It appears that to have working ASIC with the LLM baked into it we need to place and route macroblocks, and not a great variety of them. These macroblocks can be pre-placed-and-routed, available as masks already and shared between different LLMs.

Thus it appears that the tapeout delay can be substantially lower than a year.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#364

Post by Ductapemaster »

Eisenstein wrote: post #235
Ductapemaster wrote: post #232

Earlier quoted context omitted.

If you were to remove the heat at a sufficient rate by, say, turning the lid into a heat exchanger, you would have a stable system.

How is that different from not using evaporative cooling and just putting the heat exchanger on the burner?

Water is being used to get heat from one place to another. The idea is being able to separate the heat generation and the heat extraction

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#365

Post by rubzah »

ehnto wrote: post #123
corford wrote: post #120

These nascent inference chip efforts are reminding me of the early 3dfx / riva / mach / powervr days. Will be interesting to see if inference chips are here to stay and, if so, who the eventual dominant player(s) will be

Which in turn reminds me of Soundblaster audio cards! I suspect inference chips are closer to the GPU story than the Soundblaster story though. I remember one soundblaster card I bought came with a Lara Croft demo, that exploited the incredible immersion of real time dynamic reverb. Genuinely I think game audio took a few steps back from that heady era, the innovation in audio likely didn't sell as many cards as grap…

I recently heard, for the first time, what Space Quest sounded like with a Roland board attached. It was mindblowingly amazing. And all most anyone ever experienced was bleep, bloop.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#366

Post by Eisenstein »

Ductapemaster wrote: post #364
Eisenstein wrote: post #235

Earlier quoted context omitted.

How is that different from not using evaporative cooling and just putting the heat exchanger on the burner?

Water is being used to get heat from one place to another. The idea is being able to separate the heat generation and the heat extraction

You are adding an extra step though. Use a closed loop coolant and remove the heat from that coolant with the heat exchanger. Why would evaporation in between be more efficient?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#367

Post by bwfan123 »

Traster wrote: post #300

This semi-analysis article reads a lot more like an OpenAI press release than a real analysis. And to be honest some of the statements seem like just straight up lies - they initially claim they were invited to benchmark it, and then half way down switch to claiming that OpenAI provided all the numbers. This really kind of sucks, because I want to read actual detailed nuanced and credible analysis of what's happening…

> some of the statements seem like just straight up lies

Why the angst ? I suspect this announcement punctured a lot of people's bubbles, and many are in disbelief and denial and hence the emotional reaction seen here. That a company which never designed chips could suddenly leapfrog the best in the industry. What many forget is that openAI and anthropic are in a unique position to own the end-user experience, and that provides them a distinct advantage. But, making announcements and actually delivering are two different things, and it remains to be seen if these are actually viable. In any case, it gives openai leverage over their vendors.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#368

Post by kaveh_h »

NVIDIA acquired Groq and have already integrated their technology in their solution. It seems it can really increase inference performance per energy used. I don’t believe The benchmark compares Jalapenjo together with this solution. https://nvidianews.nvidia.com/news/nvidia-groq-3-lpx-now-in-...
Post reply on HN