Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

311–320 of 390 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#311

Post by fooker »

mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

Computer Science history has taught us that this decision was almost always wrong.

Every time a company has spent resources doing this, a competitor innovated on the software and made the custom hardware irrelevant.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#312

Post by porridgeraisin »

Eridrus wrote: post #174
porridgeraisin wrote: post #118

Earlier quoted context omitted.

It is being used as part of a combined system. For example AWS is pushing for Trainium + WSE 3. The WSE 3 does the decode and the Trainium does the prefill. Even in nvidia land rubin + LPU does a similar thing. It has its downsides of course - if your traffic swings prefill heavy to decode heavy, you can't suddenly use your lpu for prefill. With GPUs they're totally interchangeable. Tradeoffs.

AFAIK You can use WSE/LPU for prefill, it's just less efficient to do so.

Well ya, that efficiency is why it's split.

There is also the other idea where you run your attention layer on the GPU/TPU/Trainium and the FFN on the SRAM accelerator. Because KV cache is more difficult on cerebras etc, while MOE latency is easier to deal with

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#313

Post by imtringued »

datakan wrote: post #8
varispeed wrote: post #4

Why they don't research how to make their own RAM and they have to buy it from the common market? They should GTFO with this crap. Create barriers to computing for ordinary people while milking businesses for tokens.

People keep saying stuff like this without understanding what it takes to make RAM. It's one of, if not the most, heavily patented things in the world. The second you dip your toes into those waters the lawsuits begin. If somehow you get around the patent issues, you're now faced with huge research and development costs, fabs to build, processes to sort out and all of that has very high failure rates. Last time I che…

Nah, DRAM is easy and very regular, it's a transistor and capacitor plus a massive decoder/encoder for addressing. The hard part is that it's a crushingly low margin business and without the added AI demand there were constant boom and bust cycles wiping out the manufacturers.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#314

Post by mike_hearn »

miki123211 wrote: post #299
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

I really want this for Whisper, particularly in some kind of a power-efficient, portable form factor. I think people underestimate how much of a revolution having an always-on, privacy-preserving personal notetaker / secretary would be.

AI is bigger than just LLMs. The real place model-hardwired chips will find value is in robotics. There you need very local, low latency inferencing with relatively stable models to handle motor control and navigation tasks. Higher level reasoning can be delegated to LLMs and run asynchronously.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#315

Post by automatic6131 »

Traster wrote: post #300

This semi-analysis article reads a lot more like an OpenAI press release than a real analysis. And to be honest some of the statements seem like just straight up lies - they initially claim they were invited to benchmark it, and then half way down switch to claiming that OpenAI provided all the numbers. This really kind of sucks, because I want to read actual detailed nuanced and credible analysis of what's happening…

Semianalysis is an AI hype organisation, not a serious, unbiased semiconductor reviewer/journalist like chipsandcheese nor a documentarian of the semiconductor industry like Asianonmetry (as it relates so strongly to the modern economies of Asia). If you see something from semianalysis, you can simply ignore it.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#316

Post by amoss »

duchef wrote: post #282
g00afthrowaway wrote: post #278

Funny semi analysis has credibility here of all places. The founder is well-known in the hardware circle to be a black-market information trader. It works like this: 1. Founder befriends undergrad interns/graduate student interns, buys them gifts, invite them to dinner/yacht/house/vc parties etc, or pays them to write articles 2. Founder extracts insider information out of these interns 3. Founder sells this informat…

So you're saying the information is reliable?

Some of it is well sourced but the remainder is well sauced.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#317

Post by meowers1 »

mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

[dead]

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#318

Post by Marha01 »

basilgohar wrote: post #217
jeremyjh wrote: post #139

Earlier quoted context omitted.

I think Sol is already good enough though.

"640k (token context) should be enough for anyone."

Well, with proper compaction algorithm (like in Codex), it could indeed be useful for majority of tasks even in the future. The context length in Codex is just 272k, but it can reason well about much larger codebases due to good exploration and compaction algorithm.

I have yet to saturate the 1M context of Gemini, for example.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#319

Post by walthamstow »

miki123211 wrote: post #299
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

I really want this for Whisper, particularly in some kind of a power-efficient, portable form factor. I think people underestimate how much of a revolution having an always-on, privacy-preserving personal notetaker / secretary would be.

I don't think any underestimates how powerful it would be. It's the social aspect that is difficult. I wouldn't want to be sat in a pub with you and your always-on notetaker.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#320

Post by conradfr »

corford wrote: post #120

These nascent inference chip efforts are reminding me of the early 3dfx / riva / mach / powervr days. Will be interesting to see if inference chips are here to stay and, if so, who the eventual dominant player(s) will be

Also the PhysX cards. Not sure how long they were useful.
Post reply on HN