Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kaist develops next-generation ultra-low power LLM accelerator

en.yna.co.kr

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Kaist develops next-generation ultra-low power LLM accelerator

#21

Post by zachbee »

dartos wrote: post #3

> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?

Neuromorphic computing basically uses individual "neurons", represented with either analog or digital circuits, which communicate using asynchronous pulses called "spikes". Unlike the human brain, neuromorphic chips are 2D, but we can replicate a good amount of neural dynamics in silicon.

It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures.

https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/Ben...

Re: Kaist develops next-generation ultra-low power LLM accelerator

#22

Post by zachbee »

Neuromorphic computing is cool, but not new tech. However, using a neuromorphic spiking architecture to run LLMs seems new. Unfortunately, there doesn't seem to be a paper associated with this work, so there's no deeper information on what exactly they're doing.

Re: Kaist develops next-generation ultra-low power LLM accelerator

#23

Post by ben-schaaf »

moffkalast wrote: post #7
geuis wrote: post #6

Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/

It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.

> pure L3 cache with a whopping 230 MB per card

Just to put these numbers in perspective a desktop 8 core 7800x3d has 96MB of L3 cache, and the top-end 96-core Epyc 9684X has 1.15GB of L3.

Re: Kaist develops next-generation ultra-low power LLM accelerator

#24

Post by hackerlight »

moffkalast wrote: post #7
geuis wrote: post #6

Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/

It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.

What's the cost per inference relative to H100? Isn't that the number to care about?

Re: Kaist develops next-generation ultra-low power LLM accelerator

#25

Post by 542458 »

hackerlight wrote: post #24
moffkalast wrote: post #7

Earlier quoted context omitted.

It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.

What's the cost per inference relative to H100? Isn't that the number to care about?

If you believe the marketing material it’s lower. Their API is the cheapest around, so either it’s true or they’re subsidizing.

Re: Kaist develops next-generation ultra-low power LLM accelerator

#26

Post by hackerlight »

542458 wrote: post #25
hackerlight wrote: post #24

Earlier quoted context omitted.

What's the cost per inference relative to H100? Isn't that the number to care about?

If you believe the marketing material it’s lower. Their API is the cheapest around, so either it’s true or they’re subsidizing.

Another consideration: Even if it's slightly more expensive, that can be OK if you care about inference speed. I'd pay 50% more for GPT-4 if it could deliver results that quick.

Re: Kaist develops next-generation ultra-low power LLM accelerator

#28

Post by [deleted] »

moffkalast wrote: post #7
geuis wrote: post #6

Want to reference Groq.com. They are developing their own inference hardware called an LPU https://wow.groq.com/lpu-inference-engine/ They also released their API a week or 2 ago. Its significantly faster than anything from OpenAI right now. Mixtral 8x7b operates at around 500 tokens per second. https://groq.com/

It's not so much an accelerator as it is addressing the main inference bottleneck (i.e. memory latency) with sheer brute force by throwing money at the problem. They've made accelerators out of pure L3 cache with a whopping 230 MB per card. They cited something like 500 cards to load one single Mixtral instance, which probably cost over $10M to build. It's a supercomputer essentially.

[deleted]

Re: Kaist develops next-generation ultra-low power LLM accelerator

#29

Post by int_19h »

wmf wrote: post #17
LoganDark wrote: post #15

Earlier quoted context omitted.

They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo. I imagine Mixtral by itself would only take something like 200-300 LPUs

Only $5M then.

$5M once, upfront. But given the significantly increased throughput, how fast does that pay for itself?
Post reply on HN