Kaist develops next-generation ultra-low power LLM accelerator
1–10 of 41 posts
Re: Kaist develops next-generation ultra-low power LLM accelerator
#2> The 4.5-mm-square chip, developed using Korean tech giant Samsung Electronics Co.'s 28 nanometer process, has 625 times less power consumption compared with global AI chip giant Nvidia's A-100 GPU, which requires 250 watts of power to process LLMs, the ministry explained.
>processes GPT-2 with an ultra-low power consumption of 400 milliwatts and a high speed of 0.4 seconds
Not sure what's the point on comparing the two, an A100 will get you a lot more speed than 2.5 tokens/sec. GPT 2 is just a 1.5B param model, a Pi 4 would get you more tokens per second with just CPU inference.
Still, I'm sure there's improvements to be made and the direction is fantastic to see, especially after Coral TPUs have proven completely useless for LLM and whisper acceleration. Hopefully it ends up as something vaguely affordable.
Re: Kaist develops next-generation ultra-low power LLM accelerator
#3> New structure mimics the layout of neurons and synapses
What does that mean, practically?
How can you mimic that layout in silicon?
Re: Kaist develops next-generation ultra-low power LLM accelerator
#4> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
Re: Kaist develops next-generation ultra-low power LLM accelerator
#5> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part.
I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?
Re: Kaist develops next-generation ultra-low power LLM accelerator
#6Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/
They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/
Re: Kaist develops next-generation ultra-low power LLM accelerator
#7Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/
It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.
Re: Kaist develops next-generation ultra-low power LLM accelerator
#8Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/
It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.
[deleted]
Re: Kaist develops next-generation ultra-low power LLM accelerator
#9> The 4.5-mm-square chip, developed using Korean tech giant Samsung Electronics Co.'s 28 nanometer process, has 625 times less power consumption compared with global AI chip giant Nvidia's A-100 GPU, which requires 250 watts of power to process LLMs, the ministry explained. >processes GPT-2 with an ultra-low power consumption of 400 milliwatts and a high speed of 0.4 seconds Not sure what's the point on comparing the…
Which of the model requirements of Coral TPUs [1] are the most problematic for LLMs?
[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...
Re: Kaist develops next-generation ultra-low power LLM accelerator
#10Earlier quoted context omitted.
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part. I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discard…
But this group claims to have demonstrated a way to use SNNs to run LLMs effectively and with vastly less energy usage.