Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

221–230 of 236 posts

Re: Building a personal, private AI computer on a budget

#221

Post by wolfhumble »

sethd wrote: post #189
wolfhumble wrote: post #48

Earlier quoted context omitted.

It would be better/cheaper to buy an external Thunderbolt 5 enclosure for the NVME drive you need.

I looked into this a couple months ago and external TB5 was still more expensive at 1-2 TB not sure about above, though.

Going from 500GB to 2TB built-in is €600 in the US at the moment.

Samsung 990PRO 2TB is $170 and Acasis T5 80Gbps is €300. So it makes sense to buy external for ≥ 2TB, more flexible as well :-)

For 1TB it makes more sense to buy built-in as you note above.

Re: Building a personal, private AI computer on a budget

#222

Post by fkyoureadthedoc »

fkyoureadthedoc wrote: post #103
diggan wrote: post #86

Earlier quoted context omitted.

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit

7800X3D, 32GB DDR5, 4090:

    total duration:       10.5922028s
    load duration:        21.1739ms
    prompt eval count:    36 token(s)
    prompt eval duration: 546ms
    prompt eval rate:     65.93 tokens/s
    eval count:           467 token(s)
    eval duration:        10.023s
    eval rate:            46.59 tokens/s

Re: Building a personal, private AI computer on a budget

#223

Post by numpad0 »

GTP wrote: post #211
almostgotcaught wrote: post #207

Earlier quoted context omitted.

> Is that not the case? it is - they're laughably slow and not even supported by latest CUDA > NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.

But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?

The PTX hack is for backend runner and training infra, the public weights are often executed using existing backends. Especially R1-distill-* models are.

Re: Building a personal, private AI computer on a budget

#224

Post by therealfiona »

almostgotcaught wrote: post #212
GTP wrote: post #211

Earlier quoted context omitted.

But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?

> it isn't a big deal? friend you shouldn't make comments like this unless you understand the definitions of the words. Deepseek wrote some parts of their kernels using PTX. newsflash: PTX support for features is lockstep with CUDA support for the same features ie the fact that CUDA doesn't support it means you couldn't write the PTX to use those features either.

It is poor form to condemn someone from asking a question.

Thank you for providing the information to clear up ignorance though.

Re: Building a personal, private AI computer on a budget

#225

Post by TrueDuality »

yjftsjthsd-h wrote: post #205
axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. I thought the problem was that those cards have loads of RAM but lack really important compute capabilities such that they're kind of useless for actually running AI workloads on. Is that not the case?

I'm running P41s in one of my test boxes. These don't have support for BF16 but they do support F16 and F32 and those are accelerated to a certain degree, they're lacking kernels that are as optimized but its not terribly hard to adapt other ones for the purposes.

You don't get great out-of-the-box performance but it only took me three work days or so with no experience writing these to adapt, test, and validate a kernel using the acceleration hardware that was available (no prior experience writing these kernels).

They're not as powerful as others but still significantly better than running on a CPU alone and I'd bet my kernel is missing more advanced optimizations.

My issue with these was the power cable and fans. The author touches on the fans and I did try a 3D printed shroud and some of the higher pressure fans but I could only run the cards in short stints. I ended up making an enclosure that went straight out of the case using two high pressure SAN array fans I harvested from the IT graveyard per card and making a hole with an angle grinder.

The power cable is NOT STANDARD on these. I had to find a weird specific cable to adapt the standard 8-pin GPU connector and each card takes two of these bad boys.

Re: Building a personal, private AI computer on a budget

#226

Post by DrNosferatu »

CamperBob2 wrote: post #84
elorant wrote: post #20

Earlier quoted context omitted.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.

I tried that that: ~1.5 to 3 tokens/sec.

Re: Building a personal, private AI computer on a budget

#227

Post by almostgotcaught »

therealfiona wrote: post #224
almostgotcaught wrote: post #212

Earlier quoted context omitted.

> it isn't a big deal? friend you shouldn't make comments like this unless you understand the definitions of the words. Deepseek wrote some parts of their kernels using PTX. newsflash: PTX support for features is lockstep with CUDA support for the same features ie the fact that CUDA doesn't support it means you couldn't write the PTX to use those features either.

It is poor form to condemn someone from asking a question. Thank you for providing the information to clear up ignorance though.

this is a question:

> is deepseak's use of PTX instead of CUDA relevant here?

this is a conclusion/assumption thinly veiled as a question

> Deepseek R1 doesn't use CUDA, so ... it isn't a big deal?

note, genuine questions don't already presuppose an answer.

Re: Building a personal, private AI computer on a budget

#228

Post by drcongo »

vunderba wrote: post #193
drcongo wrote: post #128

Earlier quoted context omitted.

I use a Mac Studio for Stable Diffusion, what's special about the Mac Mini that means it won't work?

What models are you using? Stable diffusion 1.5, SDXL, or flux? I've heard that Macs are pretty slow with XL and borderline unusable for flux requiring minutes at a time to generate a single image - whereas an RTX4090 can generate a 1024x1024 image with the higher quality Flux Dev model (not schnell) in 14 seconds. OP is probably correct that if you want to branch out of just strictly LLM's, cuda is the way to go. I'…

I've used 1.5 through 3.5, XL has been kinda fine for me but tiny adjustments can take it from good to terrible. Good point about Flux though, that's awful on the Mac.

Re: Building a personal, private AI computer on a budget

#229

Post by sethd »

wolfhumble wrote: post #221
sethd wrote: post #189

Earlier quoted context omitted.

I looked into this a couple months ago and external TB5 was still more expensive at 1-2 TB not sure about above, though.

Going from 500GB to 2TB built-in is €600 in the US at the moment. Samsung 990PRO 2TB is $170 and Acasis T5 80Gbps is €300. So it makes sense to buy external for ≥ 2TB, more flexible as well :-) For 1TB it makes more sense to buy built-in as you note above.

It also depends on the quality of that enclosure and whether or not it adds heat or fan noise.

Re: Building a personal, private AI computer on a budget

#230

Post by behohippy »

Eisenstein wrote: post #217
behohippy wrote: post #214

Earlier quoted context omitted.

Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too

We are talking about dynamically quantizing KV cache, not the model weights.

I run the KV cache at Q8 even on that model. Is it not working well for you?
Post reply on HN