Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

211–220 of 390 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#211

Post by vunderba »

pantalaimon wrote: post #205
mf_tomb wrote: post #190

Earlier quoted context omitted.

"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal

Well we'll see those surplus chips being repurposed for toys then. Who wouldn't want a new Furby that can actually hold a conversation.

I've seen several attempts even on HN of the LLM meets Teddy Ruxpin (or more accurately AG Talking Bear) but most of them offloaded the AI to some off-site servers.

I’d like to think that most parents would be weary of handing their children what basically amounts to a tape recorder that siphons all the data off to a large corporation.

OTOH, a completely local one (LLM + VAD + Speech Rec) would be a fun little thing to build.

https://en.wikipedia.org/wiki/AG_Bear

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#212

Post by manquer »

ChoosesBarbecue wrote: post #2

This is most impressive. The interesting question to me, is outside of the LLM accelerator space: will generalized chips have massive leaps in performance once LLM technology is used to create the next generation? In general, will we see rapid advances while we extract the value of these models in creating architectures? I'm so far removed from the space that this is a very naive interpretation of all this, but I'm c…

ASICs always do better than general purpose chips. General purpose chips is turtles and turtles of virtualization and have to consider 4+ decades of backward compatible instructions set support.

ASICs are deployed when the application area is economically large enough to so there is return on the investment in developing one. Bitcoin mining few years ago or today inference or more mundane things like video decoding/encoding.

General purpose chips on the other hand have to be general purpose first to be useful, i.e. support as many application domains and instruction sets as possible . It can be long tail of support which both slow your chip down and also slow development down. Apple's took a long time to develop M series to be general purpose enough and still need even now software tooling like Rosetta to make say virtualization work for a good reason.

New tooling would always help and there is already lot of software emulation for developing chips today but you still need physical iterations to tap-out and have high enough yield, no LLM can help with that.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#213

Post by VirusNewbie »

simianwords wrote: post #32

How can OpenAI mass produce this chip at scale more economically than Nvidia which has experience in the supply chain and scale efficiencies to do it efficiently?

"How can OpenAI produce a LLM at scale more economically than Google, Amazon, and Microsoft which have experience in planet scale software and scale efficiencies unlike them".

One answer is they're quite good at poaching talent.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#215

Post by simianwords »

dgellow wrote: post #135
phoghed wrote: post #111

Earlier quoted context omitted.

Pointing at the nay sayers who say tokens are heavily subsidized and it’s all going to come crashing down soon, surely any moment now

I mean, it will obviously crash at some point. With so much pressure on token price to go down that means way less opportunity for margin for AI providers. OpenAI is in a pretty bad situation

What does this have to do with margins? It can remain the same once prices go down

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#216

Post by usef- »

twobitshifter wrote: post #200
mf_tomb wrote: post #190

Earlier quoted context omitted.

"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal

It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.

The whole point is that it's supposed to be more efficient. But models are also still getting absurdly more efficient every year, so you're likely nullifying much/most of the advantage. 18 months is a long time right now (and 18 is only time to tape out, not operational in data centers).

Even if the balance was net positive, you would also not be able to train them against new tools/harnesses or knowledge. How many years do you expect to keep using them?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#217

Post by basilgohar »

jeremyjh wrote: post #139
bmulholland wrote: post #97

Earlier quoted context omitted.

Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this. Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a mont…

I think Sol is already good enough though.

"640k (token context) should be enough for anyone."

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#218

Post by kimixa »

nextaccountic wrote: post #196
cherioo wrote: post #194

Earlier quoted context omitted.

Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all. The more problem like these they solve the more they will look like GPU.

couldn't one just add some hundreds of GB of HBM?

Yeah, but then there's the size of KV cache needing to be read through that HBM interface for each token, putting a hard limit on the tok/s based on the memory bandwidth.

On some models a large context can be a notable proportion of the size of the weights themselves.

For example, qwen 3.8 27b uses ~64kb/token for the kv cache - so for a 256k token context that's ~16gb of the kv cache for a ~54gb model (assuming 2 bytes-per-param/f16 for both).

So if the current non-baked-in chip is already memory bandwidth bound, as is often the case for current hardware and models, and the "only KV cache in HBM" chip has the same total memory bandwidth, it can only ever be (54/16)=~3.4x faster for the baked in-silicon model.

EDIT: I guess actually (54+16)/16=~4.3x faster, as the current implementation would need to read that KV cache too :)

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#219

Post by adventured »

twobitshifter wrote: post #200
mf_tomb wrote: post #190

Earlier quoted context omitted.

"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal

It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.

The good enough level isn't ever arriving. We're in the first or second inning for LLMs. They will rapidly subdivide in complexity, they will not stagnate in the next decade.

Beyond the model, when would you freeze processor performance, such that it was good enough? Because that's exactly what freezing on Talaas is premised around.

The semiconductor technology will also continue to improve. You lose twice. Talaas is one of the dumbest ideas I've seen in semiconductors in decades.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#220

Post by thoughtbefore »

bmulholland wrote: post #97
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this. Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a mont…

It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer?

If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.

Post reply on HN