21–30 of 236 posts
Earlier quoted context omitted.
200+ comments, https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.
Runs is an overstatement though. With 4 tokens/second you can't use it on production.
Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?
No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.
You can do 8b local on the latest iPhones.
How useful is this, though? In my modest experience, these tiny models aren't good for much more than tinkering with, definitely not something I'd integrate into my workflow since the output quality is pretty low. Again, though, my experience is limited. I imagine others know something I do not and would absolutely love to hear more from people who are running tiny models on low-end hardware for things like code assi…
Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent.
How are HN users handling this?
Great breakdown!. The "own your own AI" at home is a terrific hobby if you like to tinker, but you are going to spend a ton of time and money on hardware that will be underutilized most of the time. If you want to go nuts check out Mitko Vasilev's dream machine. It makes no sense if you don't have a very clear use case that only requires small models or really slow token generation speeds. If the goal however is not…
Not necessarily. For non-professional purposes, I've spent zero dollars (no additional memory or GPU) and I'm running a local language model that's good enough to help with many kinds of tasks including writing, coding, and translation.
It's a personal, private, budget AI that requires no network connection or third-party servers.
The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…
An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s.
The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…
So lets say we'd run a model on a Mac Mini M4 with 24GB RAM, how many tokens/s are you getting? Then if we run the exact same model but with a RTX 3090ti for example, how many tokens/s are you getting?
Do these comparisons exist somewhere online already? I understand it's possible to run the model on Apple hardware today, with the unified memory, but how fast is that really?
The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?
Combine the best of both worlds. I have a local assistant (communicate via Telegram) that handles tool-calling and basic calendar/todo management (running on a RTX 3090ti), but for more complicated stuff, it can call out to more advanced models (currently using OpenAI APIs for this) granted the request itself doesn't involve personal data, then it flat out refuses, for better or worse.
Earlier quoted context omitted.
Runs is an overstatement though. With 4 tokens/second you can't use it on production.
Isn’t 4 tps good enough for local use by a single user, which is the point of a personal AI computer?
The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…