GPU-Accelerated LLM on an Orange Pi • Hacker News Board

Re: GPU-Accelerated LLM on an Orange Pi

#81

Post by kkielhofner » Wed, Aug 16, 2023, 5:38 PM UTC

elorant wrote: post #65

cstejerean wrote: post #64

Earlier quoted context omitted.

$15k for a box with 144 GB GPU RAM is not bad, but I'm not clear on how they're going to run that from a single 1600W PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd need 2x 1600W PSUs and two separate 15amp circuits to run such a thing at home (in the US).

You can undervolt gpus without loosing that much performance.

With Nvidia this is (typically) setting the power limit[0]. Even with the default power limits of my RTX 4090s (480 watts)I don't think I've ever seen an ML workload get close to that and as the referenced article demonstrates you can more-or-less set your power limit to somewhere around 75% of max without losing much if any performance depending on workload.

It doesn't take much testing to come up with the ideal power limit for your given workload(s).

[0] - https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit...

Re: GPU-Accelerated LLM on an Orange Pi

#82

Post by kkielhofner » Wed, Aug 16, 2023, 5:42 PM UTC

jrockway wrote: post #69

elorant wrote: post #65

Earlier quoted context omitted.

You can undervolt gpus without loosing that much performance.

But realistically, the 2x PSU case was a common crypto mining setup and yeah, you have your electrician install two circuits right next to each other. Or use a 240V PSU.

Nitpick - almost all modern power supplies are auto-ranging between 100-240V. In 240V locales at 80% constant load on a 15amp circuit this is 2880 watts. In the US you would (ideally) drop a 240V circuit from the panel and make sure you have sufficient supply amperage from the utility provider.

Re: GPU-Accelerated LLM on an Orange Pi

#83

Post by kkielhofner » Wed, Aug 16, 2023, 5:59 PM UTC

sunshine-o wrote: post #28

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?

I've been bamboozled by the Jetson series over at least three generations on a variety of platforms (TX1, Nano, AGX, Orin Nano).

They have their "special place" for certain applications but the software is a mess (old driver and CUDA versions, Jetpack is still based on Ubuntu 20.04), the ARM cores are (very) weak relative to ARM flagships, and the performance/price ratio makes no sense unless you really need the form factor and energy efficiency. Oh yeah and the SD card storage is typically frustratingly slow and often unreliable.

A $500 Jetson Nano devkit with 8GB of shared RAM has roughly 10% of the performance of even ancient cards like the GTX 1070 (8GB VRAM alone) that you can throw in a random used x86_64 tower or whatever for $300 all-in. For the extra couple of hundred dollars difference you can get a more recent GPU with higher compute capability, extra storage, more system RAM, whatever. Significantly higher power usage and larger form factor but with power optimization, scaling, etc this approach makes very little difference in practice for occasional inference workloads for "typical home use". I have this configuration idling (with models loaded) at 20 watts, scaling up to around 100-150w for however long given inference loads take to execute, and then scaling back down.

Re: GPU-Accelerated LLM on an Orange Pi

#84

Post by NavinF » Fri, Aug 18, 2023, 8:18 AM UTC

chad1n wrote: post #40

sunshine-o wrote: post #28

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?

I was thinking that powerful used phones will be extremely valuable in the next years, since they are fairly cheap and more powerful than these devices.

Not gonna happen. Perf/watt on phones is terrible compared to consumer GPUs