Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

441–450 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#441

Post by otterley »

SwellJoe wrote: post #414
otterley wrote: post #386

Earlier quoted context omitted.

The question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.

The models have to run on something or they're useless. They can't run on future hardware today, and people want to use models today. So, if hardware is obsolete the day it ships, we're all using obsolete hardware, and there's no alternative to that.

Taalas encodes the model into the hardware itself. The two are inextricably coupled. It’s like buying a CNC router that can’t be reprogrammed to build anything other than a specific predetermined kitchen cabinet. And the model used inside is frozen many months before the hardware ships, since the process from tapeout to production takes that long.

In contrast, tomorrow’s models will typically run, although perhaps more slowly, on general-purpose inference hardware that was released today or even years ago.

Re: OpenAI unveils its first custom chip, built by Broadcom

#442

Post by Jyaif »

ngomez wrote: post #403
voidUpdate wrote: post #400

Earlier quoted context omitted.

> "Downside is this chip would be huuuuge - a whole wafer." Why don't we have chips like that? If a CPU the size of a postage stamp can do x amount of performance, imagine how much performance you could get if you used an entire wafer of chips running in parallel. Obviously there would be certain use cases, like you couldn't fit an entire wafer in a phone, but still

Using the space of an entire wafer for one chip would result in extremely low manufacturing yields. Even with state of the art silicon cleanrooms, there will still be defects in parts of the output. With CPUs and GPUs, chip makers can disable faulty cores and bin them as lower SKUs to get some yield out of it. But if you're using an entire wafer to embed weights, and a speck of dust causes a printing defect that make…

What's the difference between disabling faulty cores and disabling the parts of the wafer that have defects?

Re: OpenAI unveils its first custom chip, built by Broadcom

#443

Post by jsenn »

nickpinkston wrote: post #86

This is very cool to see - seems like soooo much efficiency waiting to be unlocked at the chip level. What's everyone think of Taalas? They're actually burning the LLM model into the silicon, with some onboard memory for fine-tuning. They claim huge cost / latency wins. Super fast demo live at: https://chatjimmy.ai/ https://taalas.com/ https://www.reddit.com/r/singularity/comments/1r9frzk/taalas...

Their demo is almost unbelievably fast, but as I understand it, the limitation of Taalas's strategy is KV-cache. This grows with context length, so either needs to be stored in SRAM (small) or streamed in (slow). Even for a tiny model like the Llama 8B they have in their demo, the KV cache will be ~64kb per token at 8-bit quantization, so at a 1,000-token sequence length you are already at 64MB of SRAM for a single user. This is probably why their demo only lets you generate 1,000 tokens: they can't go beyond that without slowing down inference.

So I'm curious what their strategy is. It seems to me that the options are: 1. Target smaller usecases that can live with a tiny context window 2. Use huge amounts of SRAM (at which point they look like Groq or Cerebras) 3. Make it up with extreme KV-cache compression/quantization 4. Run linear-attention/sliding window attention models

Other commenters have mentioned robotics as a potential application, which sounds interesting.

Re: OpenAI unveils its first custom chip, built by Broadcom

#444

Post by WithinReason »

londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

One token per clock cycle at 1B parameters would imply 2 ExaFLOPS, consuming about 10 KWs

Re: OpenAI unveils its first custom chip, built by Broadcom

#445

Post by DANmode »

inemesitaffia wrote: post #391
NBJack wrote: post #379

Earlier quoted context omitted.

Pretty much every example is considered an abysmal failure that often costed the actual workers their careers while their CEO carried on. If you consider that outcome a worthwhile endeavor, I don't know what else to say.

He's definitely not talking about worthy endeavour. He's talking about an endeavour reaching the market. I'm sure if Zuckerberg wants to spend $10B on Nuclear Fusion it will happen.

It’s fission, not fusion:

https://www.esgdive.com/news/meta-inks-nuclear-deals-terrapo...

…and if they do all of this, it’ll be closer to $20B than 10!

Re: OpenAI unveils its first custom chip, built by Broadcom

#446

Post by sentinalien »

WithinReason wrote: post #419
seydor wrote: post #115

Earlier quoted context omitted.

realistically, how hard are AI accelerators to design?

The hardware? Not too difficult, there are dozens of startups. The software? Only NVIDIA could do it so far sufficiently well.

How many profitable startups are there?

Re: OpenAI unveils its first custom chip, built by Broadcom

#447

Post by kccqzy »

amelius wrote: post #434
kccqzy wrote: post #299

Earlier quoted context omitted.

My hunch is that this change is driven by bean counters.

Who says Google isn't doing its own designs mostly?

Oh they definitely are. But as a transitional step, replacing Broadcom with MediaTek is probably mostly about cost.

Re: OpenAI unveils its first custom chip, built by Broadcom

#448

Post by lsaferite »

otterley wrote: post #386
jwHollister wrote: post #380

Earlier quoted context omitted.

wow if they can get something like this working, what happens to all this infrastructure? Hyperscalers have to be assuming the lifespan of that stuff wrong considering the next gen will be 1000x more efficient.

The question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.

They are betting on fast release cycles coupled with much lower costs (purchase and operations) mixed with the ability to have dynamic fine tunes on top of the static model.

Re: OpenAI unveils its first custom chip, built by Broadcom

#449

Post by groundzeros2015 »

guywithahat wrote: post #327
groundzeros2015 wrote: post #278

This is starting to sound like startup scope creep. Instead of making the AI model it’s now custom silicon, web browsers, and consumer electronics?

Maybe, but they’re also a massive company. At some point Google stopped being a startup and become a massive company with margins to look after

After they were wildly profitable.

Re: OpenAI unveils its first custom chip, built by Broadcom

#450

Post by wrsh07 »

otterley wrote: post #386
jwHollister wrote: post #380

Earlier quoted context omitted.

wow if they can get something like this working, what happens to all this infrastructure? Hyperscalers have to be assuming the lifespan of that stuff wrong considering the next gen will be 1000x more efficient.

The question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.

Right, but there exist problems that need to be routinely solved and can be solved on glm 5.2. is the model state of the art when it is published? No. But when it comes out you could optimize it and let your solver run forever for quite cheap, and that could be useful if the only problems you want it to solve (for cheap) are solvable by that model.

And the high water mark of what can be solved by open models will keep going up.

Post reply on HN