GPU-Accelerated LLM on an Orange Pi
blog.mlc.ai
GPU-Accelerated LLM on an Orange Pi
1–10 of 84 posts
Re: GPU-Accelerated LLM on an Orange Pi
#2I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything.
Perfect for preppers / survivalists.
Store it in a faraday cage along with a solar generator.
Re: GPU-Accelerated LLM on an Orange Pi
#3I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.
> knows almost everything
It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
Re: GPU-Accelerated LLM on an Orange Pi
#4I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.
> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM.
It’s less likely to hallucinate this way.
Re: GPU-Accelerated LLM on an Orange Pi
#5Nice achievement.
How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.
Re: GPU-Accelerated LLM on an Orange Pi
#6Best part is that they are using TVM.
Re: GPU-Accelerated LLM on an Orange Pi
#7Earlier quoted context omitted.
> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.
That would make a lot more sense. That way at least you have a chance to check up on the output, lest your first meal of 'Hedysarum alpinum' ends up being your last.
Re: GPU-Accelerated LLM on an Orange Pi
#8Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.
Re: GPU-Accelerated LLM on an Orange Pi
#9Earlier quoted context omitted.
> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.
It would be great to see such project implemented, I wonder how good would it perform
Re: GPU-Accelerated LLM on an Orange Pi
#10Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.
That's a tricky question. You're going to have to multiplex the use of the device, but since these are mostly 'ping-pong' style uses you can use something called a 'utilization factor' to figure out what a reasonable upper bound is where you still get an answer to your query in acceptable time. The typical mechanism is an input queue with a single worker to use the device. The cut-off is when the queue becomes unacceptably long, in which case you would have to throw an error or be content with waiting (possibly much) longer for your answer. This is usually capped by some hard limit on the length of the queue (for instance: available memory) or the fact that the queue fills up faster than that it can empty even over a complete daily cycle. Once that happens you need more hardware.