31–40 of 41 posts
Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/
It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.
Neuromorphic computing is cool, but not new tech. However, using a neuromorphic spiking architecture to run LLMs seems new. Unfortunately, there doesn't seem to be a paper associated with this work, so there's no deeper information on what exactly they're doing.
> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
Earlier quoted context omitted.
It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.
What's the cost per inference relative to H100? Isn't that the number to care about?
So unless you need the fast latency of Groq, consumer hardware seems to be a lot cheaper for the same thoughput.
Earlier quoted context omitted.
They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo. I imagine Mixtral by itself would only take something like 200-300 LPUs
Only $5M then.
Neuromorphic computing is cool, but not new tech. However, using a neuromorphic spiking architecture to run LLMs seems new. Unfortunately, there doesn't seem to be a paper associated with this work, so there's no deeper information on what exactly they're doing.
I heavily doubt that they are running LLMs on this.
Earlier quoted context omitted.
$5M once, upfront. But given the significantly increased throughput, how fast does that pay for itself?
You need computers for all of them and megawatts of power, power supplies, cooling, and power distribution.