Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-Accelerated LLM on an Orange Pi

blog.mlc.ai

Post reply on HN
Print view View original

41–50 of 84 posts

Re: GPU-Accelerated LLM on an Orange Pi

#41

Post by adrian_b »

raphaelj wrote: post #34

Any idea how the GPU compares to the Nvidia's Jetson series?

It is difficult to find information about the performance of the ARM GPUs that can be compared with that of NVIDIA/AMD/Intel.

However it seems that the Mali-G610 MC4 is in about the same range as the cheaper models of the old Jetson Xavier.

The newer Jetson Orin models have much faster Ampere GPUs with between 1024 and 2048 FP32 ALUs. Nevertheless, the various Jetson Orin models have a price between 4 times and 13 times higher than a SBC with RK3588 and 16 GB DRAM (especially all the Orin models with more than 8 GB DRAM are very expensive) and the ratio between their prices is much greater than the ratio between their performances.

Any small computer with AMD Phoenix offers a much better GPU performance per dollar than any NVIDIA Orin. The use of NVIDIA Orin is justified only when one needs a device that is qualified for an automotive environment.

Re: GPU-Accelerated LLM on an Orange Pi

#43

Post by brucethemoose2 »

jan_Sate wrote: post #8

Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.

Its not supported by TVM yet, but there is support for Qualcomm Hexagon.

You can kinda see some of the supported backends gated behind flags in the cmake file: https://github.com/mlc-ai/relax/blob/mlc/CMakeLists.txt

Re: GPU-Accelerated LLM on an Orange Pi

#44

Post by yieldcrv »

jacquesm wrote: post #3
kvetching wrote: post #2

I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.

> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.

Yes like a person

Re: GPU-Accelerated LLM on an Orange Pi

#45

Post by brucethemoose2 »

jdfedgon wrote: post #5

Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.

If you can run it as a AI Horde worker, and the home usage is sporadic, you could definitely support more than one person.

Otherwise ~1.5 tokens/s is definitely the minimum you'd want streaming tokens to a single person.

Re: GPU-Accelerated LLM on an Orange Pi

#46

Post by brucethemoose2 »

villgax wrote: post #17

I'm already getting 1.5tok on Ubuntu running on Android via UserLand w/ Llama.cpp(v2-Q4). Don't really see acceleration. If anything I need to see my phone do something actually useful at let's say 7-10toks

mlc should already be pretty fast on Vulkan

Re: GPU-Accelerated LLM on an Orange Pi

#47

Post by gmiller123456 »

jdfedgon wrote: post #5

Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.

"Scalability" and "Single Board Computer" don't really belong in the same sentence. That said, today you can get a refurbished mini PC with a lot more power, for a lot less money than the higher end SBCs. But I didn't see any info on how portable this project is to other hardware.

Re: GPU-Accelerated LLM on an Orange Pi

#50

Post by turnsout »

dsalzman wrote: post #37
yunohn wrote: post #31

Earlier quoted context omitted.

15,000$…

The apple lisa was 25k when adjusted for inflation…prices are for early adopters

Yes, and the Lisa failed, because there just weren’t enough adopters at that price.

I struggle to see any sizable market for the tinybox, but I wish them good luck.

Post reply on HN