Does the Mali GPU have shared RAM with the CPU on this device? Is this common for ARM devices?
GPU-Accelerated LLM on an Orange Pi
71–80 of 84 posts
Re: GPU-Accelerated LLM on an Orange Pi
#72Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.
"Scalability" and "Single Board Computer" don't really belong in the same sentence. That said, today you can get a refurbished mini PC with a lot more power, for a lot less money than the higher end SBCs. But I didn't see any info on how portable this project is to other hardware.
I think the biggest advantage here is that you can run it on the GPU using shared memory, which I'm not sure how widespread it is on mini PCs (at least not intel NUCs).
You could run it using OpenVINO on IntelCPUs, but the performance would probably take a hit. It would be a lot easier though since you can just use ggml.
Re: GPU-Accelerated LLM on an Orange Pi
#73Earlier quoted context omitted.
You can undervolt gpus without loosing that much performance.
But realistically, the 2x PSU case was a common crypto mining setup and yeah, you have your electrician install two circuits right next to each other. Or use a 240V PSU.
That's exactly what I imagine would need to happen, I'm just surprised that the specs are claiming all of that with a single 1600W PSU. It's the kind of thing that doesn't inspire a lot of confidence that someone has thought through the details of what they are claiming to sell as part of this pre-order.
Re: GPU-Accelerated LLM on an Orange Pi
#74Earlier quoted context omitted.
That's a tricky question. You're going to have to multiplex the use of the device, but since these are mostly 'ping-pong' style uses you can use something called a 'utilization factor' to figure out what a reasonable upper bound is where you still get an answer to your query in acceptable time. The typical mechanism is an input queue with a single worker to use the device. The cut-off is when the queue becomes unacce…
Actually many inference systems instead batch all requests within a time period and submit them as a single shot. It increases the average latency but handles more requests per unit time. (at least, this is my understanding how production serving of expensive models that support batching work)
I've done a bunch of optimization for GPU code (in CUDA) and there are typically a few bottle necks that really matter:
- memory bandwidth
- interconnect bandwidth between the CPU and GPU
- interconnect bandwidth between GPUs
- thermals and power if you're doing a good job of optimizing the rest
I don't see how a batching mechanism would improve on any of those, superficially it looks as though that would make matters worse rather than better. Can you explain where the advantage comes from?
Re: GPU-Accelerated LLM on an Orange Pi
#75Earlier quoted context omitted.
Actually many inference systems instead batch all requests within a time period and submit them as a single shot. It increases the average latency but handles more requests per unit time. (at least, this is my understanding how production serving of expensive models that support batching work)
I've done a bunch of optimization for GPU code (in CUDA) and there are typically a few bottle necks that really matter: - memory bandwidth - interconnect bandwidth between the CPU and GPU - interconnect bandwidth between GPUs - thermals and power if you're doing a good job of optimizing the rest I don't see how a batching mechanism would improve on any of those, superficially it looks as though that would make matter…
It's a latency vs. throughput tradeoff. I was surprised as well. But most GPUs can do 32 inferences in the same time as they can do 1 inference. They have all the parallel units required and there are significant setup costs that can be amortized since all the inferences share the same model, weights, etc.
https://groq.com/wp-content/uploads/2020/05/GROQP002_V2.2.pd... the "batching" section of https://docs.nvidia.com/deeplearning/tensorrt/archives/tenso... https://le.qun.ch/en/blog/2023/05/13/transformer-batching/
Re: GPU-Accelerated LLM on an Orange Pi
#76Earlier quoted context omitted.
I've done a bunch of optimization for GPU code (in CUDA) and there are typically a few bottle necks that really matter: - memory bandwidth - interconnect bandwidth between the CPU and GPU - interconnect bandwidth between GPUs - thermals and power if you're doing a good job of optimizing the rest I don't see how a batching mechanism would improve on any of those, superficially it looks as though that would make matter…
It's a latency vs. throughput tradeoff. I was surprised as well. But most GPUs can do 32 inferences in the same time as they can do 1 inference. They have all the parallel units required and there are significant setup costs that can be amortized since all the inferences share the same model, weights, etc. https://groq.com/wp-content/uploads/2020/05/GROQP002_V2.2.pd... the "batching" section of https://docs.nvidia.co…
Very interesting, thank you. I will point one of my colleagues that is busy with this stuff to these and I thank you on his behalf as well, it is exactly the kind of thing they are engaged in.
Re: GPU-Accelerated LLM on an Orange Pi
#77does mlc work for vision models? for example, the doc mentions --max-seq-len MAX_ALLOWED_SEQUENCE_LENGTH as a command line option. This seems to imply that it only accepts language models? Also, it doesn't seem to say anything about the input model's format? pytorch weights? onnx?
Yup, here's their web stable diffusion repo: https://github.com/mlc-ai/web-stable-diffusion
The input is a model (weights + runtime lib) compiled via the mlc-llm project: https://mlc.ai/mlc-llm/docs/compilation/compile_models.html
Re: GPU-Accelerated LLM on an Orange Pi
#78Earlier quoted context omitted.
Actually many inference systems instead batch all requests within a time period and submit them as a single shot. It increases the average latency but handles more requests per unit time. (at least, this is my understanding how production serving of expensive models that support batching work)
I've done a bunch of optimization for GPU code (in CUDA) and there are typically a few bottle necks that really matter: - memory bandwidth - interconnect bandwidth between the CPU and GPU - interconnect bandwidth between GPUs - thermals and power if you're doing a good job of optimizing the rest I don't see how a batching mechanism would improve on any of those, superficially it looks as though that would make matter…
I think in the case of LLM inference the main bottleneck is streaming the weights from VRAM to CU/SM/EU (whatever naming your GPU vendor of choice uses).
If you're doing inference on multiple prompts at the same time by doing batching, you don't take more time in streaming. But each streamed weights gets used for, say, 32 calculations instead of 1, making better use of the GPU's compute resources.
Re: GPU-Accelerated LLM on an Orange Pi
#79Re: GPU-Accelerated LLM on an Orange Pi
#80Earlier quoted context omitted.
Lessons learned on the Lisa were applied to the Mac for less money to sell to more people. I suspect the same thing will happen with their VR/AR/XR headset.
They only got a second crack at it because they were floating on sales of the Apple II. If the Lisa had been their first product, we'd be saying "Apple who?" The Vision is a whole different universe… with their cash and position they could (and may) take 10 cracks at it.
Doesn't make the tactic any more or less relevant...let the people with money and motivation become the first real-world usability testers, then optimize for the rest of the population.