Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Jalapeño: Better than Nvidia Blackwell

newsletter.semianalysis.com

Post reply on HN
Print view View original

191–200 of 389 posts

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#191

Post by greenknight »

Aurornis wrote: post #182
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

> I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon. Taalas needed a giant chip (6nm) for an 8B model. At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.

Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#194

Post by cherioo »

greenknight wrote: post #191
Aurornis wrote: post #182

Earlier quoted context omitted.

> I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon. Taalas needed a giant chip (6nm) for an 8B model. At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.

Nope. But we are hitting some pretty impressive levels with 128B models. The other thing is, a lot of the time, model performance is improved with more 'thinking' time. The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all.

The more problem like these they solve the more they will look like GPU.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#195

Post by bjourne »

The article is a bit naive:

> However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use speculative decoding. Speculative decoding leads to a ~3-5x reduction in cost per token. When speculative decoding is implemented on Jalapeño, this will enable Jalapeño to serve tokens even more cost effectively.

How much speculative decoding improves throughput is workload-dependent. Yes, it can improve performance by 5x, but it can also slow down performance by 2x. It also requires a vastly more complex decode loop than the standard token-by-token decode. The tech journos didn't ask themselves if speculative decoding improves perf so much why wasn't it on by default? Perhaps, because speculative decoding is not optimal for OpenAI's ASIC? Perhaps that is also why they were only able to benchmark the not-very-representative single-user-decode case?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#196

Post by nextaccountic »

cherioo wrote: post #194
greenknight wrote: post #191

Earlier quoted context omitted.

Nope. But we are hitting some pretty impressive levels with 128B models. The other thing is, a lot of the time, model performance is improved with more 'thinking' time. The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all. The more problem like these they solve the more they will look like GPU.

couldn't one just add some hundreds of GB of HBM?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#197

Post by striking »

anukin wrote: post #178
htrp wrote: post #107

Earlier quoted context omitted.

etched tried this.... it didn't go very well

I would assume asic based llm would work really well. Why did it not go well?

https://chatjimmy.ai/ runs Llama 3.1-8B on an ASIC as a demo by https://taalas.com/ I believe.

That's quite a few parameters shy of today's trillion-weight behemoths, but it is fast.

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#198

Post by senordevnyc »

subtlejellyfish wrote: post #180
jimmySixDOF wrote: post #11

I love how now you have to consider the possible s** posting motivation behind analysis of a trillion dollar industry being conducted at a world-class level by a bunch of ex Reddit and 4Chan adjacent mods -- it's one of the best stories in AI that SemiAnalysis is not cut from the same cloth as Gartner McKinsey et al

The "industry news and research" part of the AI industry feels very... suspect to me. My intuition is telling me that it's a bunch of people with influencer-y type social media skills and no actual credentials just grifting because there's so much money floating around.

What credentials do you need to write a substack about an industry so it’s not grifting?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#199

Post by cactusplant7374 »

jrflo wrote: post #31
anthonypasq wrote: post #3

Continued hardware improvements really make it hard for me to believe token prices will not continue to plummet.

This may just be a classic case of Jevons paradox: https://en.wikipedia.org/wiki/Jevons_paradox In short, better hardware will drive down token cost in the near-term, but will drive up the demand for tokens as it gets cheap enough for other sectors to start to use it heavily. It comes from steam engines where economists originally thought that coal demand would plummet with more efficient engines, but it actually jus…

It is incredibly cheap now. What sectors are you thinking of?

Re: OpenAI Jalapeño: Better than Nvidia Blackwell

#200

Post by twobitshifter »

mf_tomb wrote: post #190
mchusma wrote: post #85

I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves. For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough. While 2 years ago nothing was useful more than 1 year long, there are many older m…

"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal

It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.
Post reply on HN