They tested on spark model, i bet it's a mix of that with focus on inference speed. Whatever it is, hopefully it shows up with current models as faster. Token/s is as big thing as anything else, and thats where they can really gain some edge over the competition.
OpenAI unveils its first custom chip, built by Broadcom
251–260 of 496 posts
Re: OpenAI unveils its first custom chip, built by Broadcom
#252Earlier quoted context omitted.
The hardware description languages (HDL) used in chip development are like programming languages. The existing models understand them and can do a lot with them. You don’t need to have separate, specialty models designed for this work to use LLMs in chip design workflows. Design verification also involves a lot of traditional programming which benefits from LLMs. So it’s not meaningless at all. You could download som…
Most HDL code is locked up behind corporate firewalls and not available as training data. While LLMs can handle it to an extent there's a lot of room for improvement. I'll bet that OpenAI and their competitors are racing to license this IP from major hardware vendors in order to compete in the chip design vertical.
Does it work better when using compiler based ecosystem (e.g. https://github.com/llvm/circt)
Re: OpenAI unveils its first custom chip, built by Broadcom
#253I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…
firmware upgrade would mean flashing a huge BIN file.
Re: OpenAI unveils its first custom chip, built by Broadcom
#254Earlier quoted context omitted.
>If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. Even for a company’s first design?
I don't think you get the newcomer novelty buff when your val approaches 13 digits.
Big companies are lumbering behemoth, crude assemblages of barely cobbled-together incentives and principal agent problems in a trenchcoat. Getting them to change direction, or worse, try something new at scale, is a massive undertaking
Re: OpenAI unveils its first custom chip, built by Broadcom
#255Earlier quoted context omitted.
17K tok/s is approaching realtime motor cortex needs for a robot with ~12 actuators (bipedal humanoid) and an IMU. I don't know how many parameters a motor cortex would need but 8B feels like it is within 2 orders of magnitude.
this is an LLM, not a motor cortex. it will output commands as text (json, ...), so comparing size is not very meaningful, especially considering neurons are highly complex and likely requires thousands of artificial simple neurons (weight+bias)
There's nothing about Taalas that is specific to an LLM
Re: OpenAI unveils its first custom chip, built by Broadcom
#256I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…
this appeared some time ago, https://taalas.com/ , but I'm sure there's others thinking these same thoughts. this would be best for small models imo, nothing frontier because that changes too fast
you can try it out here: https://chatjimmy.ai/
Re: OpenAI unveils its first custom chip, built by Broadcom
#257Earlier quoted context omitted.
Yeah, "first chip" here likely means they contracted Broadcom (or a firm with similar experience) to do all the heavy lifting. Building out your own in-house teams for this sort of thing is a decade-long project - just look how much inside Apple's early chips was licensed ARM / PowerVR cores
Apple didn't have the talent in-house until they bought Intrincity who worked with Samsung on Apple's earlier Arm chips as well. https://en.wikipedia.org/wiki/Intrinsity
I think the folks at PA Semi had some chops too.
Re: OpenAI unveils its first custom chip, built by Broadcom
#258I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…
“ Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.” Brain science people “love” traumatic brain injury cases because it can help explore what happens when bits of the “brain wafer” get damaged. We’ve learned a lot from such things. I wonder if people are intentionally “destroying” parts of the model weights to learn more about what happens? Like could you…
Somehow related:
Re: OpenAI unveils its first custom chip, built by Broadcom
#259Earlier quoted context omitted.
i don't understand what the second paragraph is saying.
In very crude terms, AFAICT , if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits thi…
> IDK how the custom hardware exploits this; would love to hear any ideas!
You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2].
[1]: https://inria.hal.science/hal-04689673/document
[2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...
Re: OpenAI unveils its first custom chip, built by Broadcom
#260[dead]