Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models is good now

vickiboykis.com

Post reply on HN
Print view View original

581–590 of 651 posts

Re: Running local models is good now

#581

Post by iwontberude »

VBprogrammer wrote: post #572
munksbeer wrote: post #556

Earlier quoted context omitted.

When discussing this, may I ask (I know you are probably bored of the actual arguments), what does "trained models on data that wasn't theirs" actually mean in practice? Again, I know these arguments have been done to death, but every human who reads source code that wasn't written by them, or views art that wasn't created by them, and practices against this art, is training their brain on data "that wasn't theirs".…

In general humans don't have perfect recall. Even people with what we might call a photographic memory don't have the ability to memorise millions of lines of code and output them with little effort. It hinges somewhat on the concept of how much you believe things are being learned and how much is just pattern matching and borrowing a solution from memory. Certainly in the early days of Copilot it was possible to get…

If it outputs copyrighted material, which it does handily, then it doesn’t really matter.

Re: Running local models is good now

#582

Post by adam_patarino »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

I always find it amusing when people would rather spend $200 / mo than let their laptop fan turn on.

Re: Running local models is good now

#583

Post by maldie »

mathisfun123 wrote: post #173
eek2121 wrote: post #169

Earlier quoted context omitted.

Not really, Qwen 27b offloads to a decent gaming GPU (RTX 4090 in my case) without needing tons of RAM.

can you give more info? llama.cpp vs vllm? config? i wanna try specifically this model

llama.cpp to get 115 tok/s on RTX 4090 with Qwen3.6-27B. For example in Windows the latest CUDA variant llama-b9678-bin-win-cuda-13.3-x64.zip and Unsloth UD-Q4_K_XL MTP gguf:

llama-server.exe --host 0.0.0.0 --alias "Qwen3.6-27B-MTP" -m "F:\Qwen3.6-27B-UD-Q4_K_XL-MTP.gguf" -c 75000 -ngl 99 --metrics --temp 0.6 --top-p 0.95 --min-p 0.00 --top-k 20 --presence-penalty 0.0 --no-mmap -t 16 --spec-type draft-mtp --spec-draft-n-max 3 --reasoning on -fa on --parallel 1 -lv 4

Note that this does not use kv cache quants as in my case quants offload to CPU and tanks performance. Also keep in mind this almost maxes VRAM usage so any additional browsers or other programs that use VRAM should be closed. For chat go to http://localhost:8080/ and minimize the window to maximize perf as the web page UI draw itself consumes a lot of GPU perf via constant context switching.

Can try bigger than -c 75000 until perf gets lower than 100 tok/s - that means something is off as windows starts paging out memory or other issues. -c 50000 seems sweetspot if running browsers and stuff that consume 2GB VRAM. If wanting more than -c 140000 then likely need to use a bit smaller model quant.

CPU usage should be near zero, maybe 1 core load. If you see 8+ core load then settings are off and something is offloaded to CPU (for example kv cache). GPU load should be about 100%, meaning it utilizes work optimally in this case.

-t 16 can be omitted or set to the amount of physical cores, not important in this dense model that is 100% in GPU.

Can be pushed to 125 tok/s with that model if using --spec-draft-n-max 4 but VRAM usage also increases, so context needs to be smaller.

If speed is not important and want max context length then remove the draft-mtp parameters and also might need to use k and v quants like --cache-type-v q8_0, leave k f16 if possible to keep quality.

Re: Running local models is good now

#584

Post by an0malous »

embedding-shape wrote: post #8

Show us the resulting code of using them! :) I want to use local models, I have the hardware for it, but while trying them out as replacements for GPT 5.5 xhigh or Opus or other SOTA models, they aren't quite ready to be replaced yet, sadly. The quality and bumps they encounter just slows down the workflow so much, even screwing up tool call syntax sometimes. But, for smaller more well-defined workflows, or as straig…

No one ever shows the resulting code of using frontier models either. Curious.

Re: Running local models is good now

#585

Post by restlake »

my most mind blowing recent development here was testing the Gemma 4 models at release for vision and image recognition vs some benchmarks I had from using Gemma 3 for the same tasks. Gemma 4 is significantly faster and massively more accurate, to a level where I fundamentally can finally turn off my wifi and run a batch of my photos through the local model and trust the results for the extensive classification that Gemma seems well suited to handle. incredible times for local LLMs

Re: Running local models is good now

#586

Post by sermakarevich »

I am running an experiment with local qwen3.6:36B for a week: https://news.ycombinator.com/item?id=48520757

It really is better than I would expect it to be. But it requires a special treatment. Since the model is smaller it needs a smaller and simpler tasks. I use smarter model to decompose the task into primitive subtasks, write good description, submit to worker with qwen3.6, review completion and create new task to fix if required (20% of cases). This workflow works fine.

Re: Running local models is good now

#587

Post by segmondy »

c0rruptbytes wrote: post #104

I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder…

I run 27B at Q8 with fp16 KV cache at 50tk/sec on 2 3090s. Not 4090, Not 5090. 6 years old GPUs.

Re: Running local models is good now

#590

Post by verdverm »

simonw wrote: post #92
chrisweekly wrote: post #72

Earlier quoted context omitted.

Obtaining that 64GB RAM is a meaningful obstacle for many.

I'm still amazed that you can run LLMs of this quality on a machine that costs less than $3,000. I used to assume that anything GPT-4 equivalent or higher would need $30,000+ of server-class hardware. That said... gemma-4-12b-qat is 7.15GB on disk so should run reasonably well in 16GB, that takes it down to MacBook Air territory https://lmstudio.ai/models/google/gemma-4-12b-qat

Second this notion. After picking up an OEM Spark and running qwen36moe/dense, I was thoroughly impressed with what such small models can do and the (reasonable) speeds you can get. I'm back to using open weight models via an API (wanted more capability for the time being), but will be getting more hardware soon (re: ds4-flash and the fable shot heard round the world)
Post reply on HN