91–100 of 208 posts
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
Earlier quoted context omitted.
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…
> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
Earlier quoted context omitted.
> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
> It's very possible that it costs you more than a cloud mode would ...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.
If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.
Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP.
From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
Earlier quoted context omitted.
> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
If you buy the computer specifically for inference it is more expensive than cloud, but if you had it anyway it's free.
Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension. --- Qwen3.8-27B-4bit, Prompt Processing (PP) 66.3 tok/s, Token Generation (TG) 11.8 tok/s Ornith-1.5-35B-A3B-MLX-4bit, PP 379.7, TG 45.8 Ornith-1.5-35B-A3B-M…
I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP. From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB…
In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible With all due respect, I'm not clear why you are so surprised ? By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ? Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=c...