181–190 of 236 posts
I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…
Earlier quoted context omitted.
On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit
> gemma2:27b What quantization are you using? What's the runtime+version you run this with? And the rest of the settings? Edit: Turns out parent is using Q4 for their test. Doing the same test with LM Studio and a 3090ti + Ryzen 5950X (with 44 layers on GPU, 2 on CPU) I get ~15 tokens/second.
Earlier quoted context omitted.
On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit
On a 3090 (24gb vram), same prompt & quant, I can report more than double the tokens per second, and significantly faster prompt eval. total_duration: 10530451000 load_duration: 54350253 prompt_eval_count: 36 prompt_eval_duration: 29000000 prompt_token/s: 1241.38 eval_count: 460 eval_duration: 10445000000 response_token/s: 44.04 Fast prompt eval is important when feeding larger contexts into these models, which is re…
> Another important finding: Terry is by far the most popular name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are loving alliteration. Mode-collapse. One reason that the tuned (or tuning-contaminated models) are bad for creative writing: every protagonist and place seems to be named the same thing.
Couldn't you just up the temperature/change some other parameter to get it to be more random/"creative"? It wouldn't be active/intentional randomness/novelty like what a human would do, but at least it shouldn't generate exactly the same naming.
Earlier quoted context omitted.
You pay a premium to get the theoretical local privacy and reliability of hosting your own models. But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.
With something like OpenRouter, you don't even have to manually integrate with multiple vendors
Earlier quoted context omitted.
Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"
When you run that, what quantization do you get? The library website of Ollama ( https://ollama.com/library/gemma2:27b ) isn't exactly a good use case in surfacing useful information like what the default quantization is.
Earlier quoted context omitted.
You probably won't be running fp16 anything locally. We typically run Q5 or Q6 quants to maximize the size of the model and context length we can run with the VRAM we have available. The quality loss is negligable at Q6.
But the inference doesn't necessarily run at the quant precision.
Earlier quoted context omitted.
Runs is an overstatement though. With 4 tokens/second you can't use it on production.
I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.
This is the problem.
If your use case is getting a small handful of non-urgent responses per day then it's not a problem. That's not how most people use LLMs, though.
Earlier quoted context omitted.
For sure and the Mac Mini M4 Pro with 64GB of RAM feels like the sweet spot right now. That said, the base storage option is only 512GB, and if this machine is also a daily driver, you’re going to want to bump that up a bit. Still, it’s an amazing machine for under $3K.
It would be better/cheaper to buy an external Thunderbolt 5 enclosure for the NVME drive you need.
The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consu…
The 1.58-bit DeepSeek R1 dynamic quant model from Unsloth is no joke. It just needs a lot of RAM and some patience.