How the new Raspberry Pi AI Hat supercharges LLMs at the edge
blog.novusteck.com
How the new Raspberry Pi AI Hat supercharges LLMs at the edge
1–10 of 12 posts
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#2I was expecting to see how they deploy, maximum model size and tokens/s.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#3"cloud" has always been vague. But "edge" has become so wishy-washy, at best meaning "not cloud, but sort of", I consider its journalistic use as incompetence.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#4I was expecting to see how they deploy, maximum model size and tokens/s.
Answer: smol, and a fraction.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#5I think the new AI hats are awesome and the TOPS of 37M is amazing.
But this article is poor. Especially later part of the article that lists the benefits of the AI accelerator reads like it was written by ChatGPT because it has a formal tone, it is wordy and repeats basics facts already covered in the article.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#6Not to be disparaging but this reads itself like it was written/padded out by an LLM.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#7"cloud" has always been vague. But "edge" has become so wishy-washy, at best meaning "not cloud, but sort of", I consider its journalistic use as incompetence.
"cloud" - somebody else's computer.
"edge" - it's like embedded, but with 5 layers of abstraction and abysmal performance.
hth.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#8Not to be disparaging but this reads itself like it was written/padded out by an LLM.
My thoughts exactly. I am glade I am not the only one who recognizes ChatGPT voice in the article.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#9This article is AI generated, and they didn't even fact check it.
An AI module like this can help a lot with processing for certain types of neural networks, but LLMs are not one of them.
LLM inference is basically bottlenecked by RAM bandwidth and how much RAM you have. Every token to be generated needs to iterate over the whole model, pulling it piece by piece from the RAM to the CPU, where some relatively small calculations are applied.
Having a separate NPU like this connected via PCIE makes LLMs much slower, since you're bottlenecked by a PCIE 3.0 x1 connection instead of your full memory bandwidth.
Re: How the new Raspberry Pi AI Hat supercharges LLMs at the edge
#10I was expecting to see how they deploy, maximum model size and tokens/s.
Answer: smol, and a fraction.
The 1B to 3B models should be runnable on Raspberry PI 5 with 8GB. That seems reasonable. Probably equivalent to the new Apple Intelligence that runs locally on device in terms of performance if you can use the same model.