Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local LLMs offline on a ten-hour flight

deploy.live

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Running local LLMs offline on a ten-hour flight

#71

Post by slowmovintarget »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

Task-driven repo, clear your context (restart the harness), check the results.

Don't try for a rambling session where you let the thing grind for hours on a huge system. It will predictably choke or end in those loops. But do a few small chunks of work, exit the harness, then pick up the next few small chunks... It doesn't feel as magical, but it seems to be more effective, even when your model is Claude.

Re: Running local LLMs offline on a ten-hour flight

#72

Post by jonaustin »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

Try https://github.com/jundot/omlx

I've been using qwen3.5 (122b) with claude code for months, and it's definitely dumber than sonnet/opus, but it works through things reasonably well (i.e. writes half-decent code and tool calls usually work), and I pretty much never run into loops now.

and make sure you're following Unsloth's recommendations for temperature/etc.

Re: Running local LLMs offline on a ten-hour flight

#73

Post by jonaustin »

fy20 wrote: post #70
sleepyeldrazi wrote: post #61

Earlier quoted context omitted.

I have been testing and using Qwen3.6 27B (running from my 3090) since it dropped and I genuinely think this is the first consumer hardware-grade model that can actually replace frontiers for a lot of workloads. I ran 8 tests on a variety of open-weights models, and opus 4.7 (1mil ctx version) and the little dense model was right behind it: https://github.com/sleepyeldrazi/llm_programming_tests/tree/... Of note is th…

Running it on a Macbook Pro M5 48GB: -hf unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL \ -c 128000 \ --parallel 1 \ --flash-attn on \ --no-context-shift \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence_penalty 0.0 \ --reasoning on \ --jinja \ --chat-template-kwargs "{\"preserve_thinking\": true}" \ --spec-type ngram-simple \ --draft-max 64 \ --timeout 1800 Maybe so…

https://github.com/jundot/omlx

note: 27b is going to be slow; use the 35b MoE if you want decent token/sec speed.

Re: Running local LLMs offline on a ten-hour flight

#75

Post by bilekas »

zahlman wrote: post #58
bilekas wrote: post #17

Trying LLM in the air with a 6.200 EUR laptop... Sorry if it's not exactly relatable..

You got downvoted, but my reaction was much the same honestly.

I don't mind being downvoted, if we all had the same opinion it would be quite boring around here. I guess I'm just jaded from AI news.

Re: Running local LLMs offline on a ten-hour flight

#76

Post by 0xffff2 »

zrail wrote: post #57
Der_Einzige wrote: post #54

Earlier quoted context omitted.

Obese people (250lb+) shouldn't even be allowed in Economy.

Sweet, free upgrades! Edit to be slightly less obtuse: surely you're not implying that a common carrier be allowed to discriminate based on facts about a passenger's body without making reasonable accommodations. Surely you're not implying that obese people not be allowed to fly at all. Surely you're not suggesting that fat people should just remove themselves from society so you don't have to deal with them. Therefo…

I'll imply those things. If you don't fit in the seat, you should have to buy two seats is a not very controversial opinion on the internet IMO. I think that opinion basically violates all of your "surely"s already.

Where do you draw the line? A 250lb person probably mostly fits in their seat still, but at some point a person is just physically going to take up two seats. Do you really think the airline should be responsible for flying them in business class (premium economy doesn't give you more width on most/all airlines)? Does it matter if their weight is due to a medical condition or just laziness? What if they're so big that even a first class seat won't contain them?

Re: Running local LLMs offline on a ten-hour flight

#78

Post by proxysna »

Der_Einzige wrote: post #55
proxysna wrote: post #9

Earlier quoted context omitted.

You need to set sampling parameters for the llm. Had the same issue with Qwen3.5 when i first started. You can grab them off the model card page usually. From Qwen3.6 page: Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalt…

min_p author here. min_p is strictly better than top_p and top_k. The big labs don't know shit about sampling, and give absolutely nuts recommendations like this. set min_p to like 0.3 and ignore top_p and top_k and you'll be fine. There's better samplers now like top N sigma, top-h, P-less decoding, etc, but they're often not available in your LLM inference engine (i.e. vLLM)

Cool, i am mostly a plumber for these things, but do you have any sort of reading that i can go through to wrap my head around it to some degree?

Re: Running local LLMs offline on a ten-hour flight

#79

Post by sleepyeldrazi »

fy20 wrote: post #70
sleepyeldrazi wrote: post #61

Earlier quoted context omitted.

I have been testing and using Qwen3.6 27B (running from my 3090) since it dropped and I genuinely think this is the first consumer hardware-grade model that can actually replace frontiers for a lot of workloads. I ran 8 tests on a variety of open-weights models, and opus 4.7 (1mil ctx version) and the little dense model was right behind it: https://github.com/sleepyeldrazi/llm_programming_tests/tree/... Of note is th…

Running it on a Macbook Pro M5 48GB: -hf unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL \ -c 128000 \ --parallel 1 \ --flash-attn on \ --no-context-shift \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence_penalty 0.0 \ --reasoning on \ --jinja \ --chat-template-kwargs "{\"preserve_thinking\": true}" \ --spec-type ngram-simple \ --draft-max 64 \ --timeout 1800 Maybe so…

I haven't honestly dug around to figure out if there's a hardware reason for it, but prompt processing has always been a lot slower for me on macs in general. I mostly use MLX on my 24GB M4 Pro though, so I will pull llama.cpp on it as well to see what the prefill is like.

I've gotten around 16 t/s gen with 4bit and mxfp4 on that model for generation. The 3090 I mentioned has a little over 900 gb/s, while those macs i think are around 270 GB/s. If my understanding is correct, macs do utilize the bandwidth better in this case, but it still doesn't make up the difference (on the 3090 it's around 30-35 t/s depending on size of ctx).

Also, do run a quick experiment removing the cache quants if you want to tinker with it a bit more, iirc KV quant does add a small overhead during prefill.

I would be very interested to know your prefill and generation numbers.

Re: Running local LLMs offline on a ten-hour flight

#80

Post by rootusrootus »

0xffff2 wrote: post #76
zrail wrote: post #57

Earlier quoted context omitted.

Sweet, free upgrades! Edit to be slightly less obtuse: surely you're not implying that a common carrier be allowed to discriminate based on facts about a passenger's body without making reasonable accommodations. Surely you're not implying that obese people not be allowed to fly at all. Surely you're not suggesting that fat people should just remove themselves from society so you don't have to deal with them. Therefo…

I'll imply those things. If you don't fit in the seat, you should have to buy two seats is a not very controversial opinion on the internet IMO. I think that opinion basically violates all of your "surely"s already. Where do you draw the line? A 250lb person probably mostly fits in their seat still, but at some point a person is just physically going to take up two seats. Do you really think the airline should be res…

The issue is for the airline to solve, since they are the ones trying to make seats comically small.

Also, you have to include other attributes. E.g. Not my problem that you have freakishly long legs, if you have to prevent me reclining then maybe you should have to pay for premium economy. And what if you are broad shouldered? Same deal, not my problem, you have to stay inside the boundaries of your own seat.

I would rather we used regulation to make economy seats a bit larger. Call it a safety issue, since it is.

Post reply on HN