Doesnt an NVIDA Spark solve most of these problems? (at 5K)
NVidia Spark is much slower (low memory bandwidth)!
Jamesob's guide to running SOTA LLMs locally
111–120 of 193 posts
Re: Jamesob's guide to running SOTA LLMs locally
#112why in gods name is a RTX PRO 6000 $13,000? supply and command?
exactly: high demand!
Re: Jamesob's guide to running SOTA LLMs locally
#113Bitcoin is so dead that jamesob is posting about AI.
Re: Jamesob's guide to running SOTA LLMs locally
#114Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too. They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished. I've read that Apple has plans once the RAM bottleneck passes to offer more R…
> once the RAM bottleneck passes Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.
It'll probably take a few years. There's many fabs under construction.
One thing holding back capacity expansion is that a lot of people are concerned this is a bubble. They're worried it'll pop and leave them with orphaned assets if they over-invest in production.
Of course maybe they're right and that will happen. If the data center construction boom ends, RAM prices will fall.
Re: Jamesob's guide to running SOTA LLMs locally
#115Earlier quoted context omitted.
> once the RAM bottleneck passes Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.
Micron have sold RAM for the next 4 years at current prices, so there are buyers expecting this to stay the same.
That means buyers have basically purchased options. If the price falls, they're underwater a little, but if the price spikes it protects them.
People do that all the time, and sometimes it doesn't pay off.
Re: Jamesob's guide to running SOTA LLMs locally
#116Earlier quoted context omitted.
I have an M5 MacBook Pro and I also have a separate GPU setup for running models. The difference in speed is significant. It's not just token generation speed, but time to first token (prompt processing). The M5 hardware is amazing for what it is, but GPUs are still so much faster. Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.
What is your GPU setup?
tinygpu kernel driver
Re: Jamesob's guide to running SOTA LLMs locally
#117Bitcoin is so dead that jamesob is posting about AI.
Bitcoin booster -> AI slopper pipeline
Re: Jamesob's guide to running SOTA LLMs locally
#118Earlier quoted context omitted.
The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Als…
The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second
I benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent,
2x3090 (has an nvlink bridge though it didn't seem to matter hugely for inference)
Qwen 3.6 27b int4: Concurrency 1: 68 tok/s output Concurrency 32: 363 tok/s output Prompt processing speed: 1520 tok/s
Qwen 3.6 35ba3b int4: Concurrency 1: 150 tok/s output Concurrency 32: 1083 tok/s output Prompt processing speed: 4324 tok/s
Macbook Pro m3 36gb RAM: Qwen 3.6 27b int4: Concurrency 1: 18 tok/s output didn't measure the other metrics and it was a slightly different benchmark.
Re: Jamesob's guide to running SOTA LLMs locally
#119Earlier quoted context omitted.
You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.
No? Any M5 Max with the upgraded GPU has the full bandwidth, which includes the 48GB model the original poster mentioned. Same as the M4 Max, where only the trimmed part had a lower bandwidth. Why are you throwing in extra cost for something thats not necessary? I know multiple people with 128GB Macs and none of us upgraded the storage. Especially not on a Studio (which isn't currently available). I will say that the…
I made a mistake, there is a $5k config with high memory bandwidth. The Max chip has two tiers (I incorrectly thought the tiers were based on memory capacity), you need the higher tier Max GPU upgrade (+$300) to get the 614 GB/s memory bandwidth but you don't need to upgrade the RAM to 128 GB to get the full memory bandwidth. So to get the 614 GB/s you need to upgrade to the max chip + upgraded GPU, but you can spec it at only 48gb if you want. So the total for an m5 max with 614 GB/s memory bandwidth is $4999-$9999 depending on config.
Still 3x lower memory Bandwidth than a dual 3090 setup which you can build for $3k with parts from facebook marketplace and run in your basement.
Re: Jamesob's guide to running SOTA LLMs locally
#120There also exists an in-between possibility, that is, if you get 128GB of vram (there are now multiple options in the market to get that amount with a unified memory architecture) you can run DeepSeek V4 flash at good speed via DwarfStar. I'm not going to spend money on this, but my gut feeling is that this would be the right compromise for a lot of people.
I just started using it on an m4 max 128 and it's the first time since buying the machine a year ago that it feels like local llm "just works" for reasonably decent coding.
Use pi though; claude code has way too much bootstrap context; slows everything way down.