Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local LLMs offline on a ten-hour flight

deploy.live

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Running local LLMs offline on a ten-hour flight

#61

Post by sleepyeldrazi »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

I have been testing and using Qwen3.6 27B (running from my 3090) since it dropped and I genuinely think this is the first consumer hardware-grade model that can actually replace frontiers for a lot of workloads.

I ran 8 tests on a variety of open-weights models, and opus 4.7 (1mil ctx version) and the little dense model was right behind it: https://github.com/sleepyeldrazi/llm_programming_tests/tree/... Of note is that opus was the only model to push back against the spec on the hardest challenge, saying 'thats not possible', when there are links in the spec to examples of it being done.

There may be problems with the mlx versions, as i haven't had any looping in all the testing i've done, which is all my agentic and coding work the last couple of days (since it dropped). I have had tool_call misses 4 or 5 times so far, which isn't ideal but no looping. First I used it in pi-mono and later when i realized it's a serious model switched to opencode.

My setup is llama.cpp running on a 3090 in WSL, unsloth IQ4_NL with those flags: --ctx-size 128000 \ --jinja \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 0.0 \ --threads 12 \ --gpu-layers 99 \ --no-warmup \ --no-mmap \ -fa on

Re: Running local LLMs offline on a ten-hour flight

#62

Post by usagisushi »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

If the "loop" you mean is the infinite reasoning cycle ("Wait, actually... On second thought..."), you might want to try setting a reasoning budget. For llama.cpp, use `--reasoning-budget 1024 --reasoning-budget-message "Proceed to final answer."` to force the model to reach a conclusion.

I admit I sometimes get caught up in the tooling for its own sake, but I find local models useful for specific tasks like migrating configuration schemas, writing homelab scripts, or exploring financial data.

It might sound a bit paranoid, but privacy is another major driver for me. Keeping credentials and private information off cloud services is worth the extra friction.

Re: Running local LLMs offline on a ten-hour flight

#63

Post by cube00 »

MacBook cable: 94W delivered

Return flight will test this with the correct cable. I expect at least 16% improvement against the 70W cap

Some plane sockets cut out completely if you attempt to draw more then the limit rather then continuing to provide power at the limit.

Re: Running local LLMs offline on a ten-hour flight

#64

Post by davidcox143 »

cube00 wrote: post #63

MacBook cable: 94W delivered Return flight will test this with the correct cable. I expect at least 16% improvement against the 70W cap Some plane sockets cut out completely if you attempt to draw more then the limit rather then continuing to provide power at the limit.

The MBP also throttles surprisingly easily. I have a 16 MBP and ended up buying a cooling stand that uses a 20w peltier cooler (solid state heat pump). It’s fixed the throttling completely, although I’m somewhat nervous about forgetting to turn the cooler off and creating condensation inside the case…

Re: Running local LLMs offline on a ten-hour flight

#65

Post by dgacmu »

bilekas wrote: post #17

Trying LLM in the air with a 6.200 EUR laptop... Sorry if it's not exactly relatable..

one one hand, I'm typing this on a 5 year old M1 MBP that I still can't bring myself to replace because it just continues to do the things I need;

on the other hand, $6200 every few years is pretty tiny compared to a typical US developer salary, so is this really that crazy if it's your primary work machine?

Re: Running local LLMs offline on a ten-hour flight

#66

Post by 0x1ch »

Der_Einzige wrote: post #54
ryandrake wrote: post #20

Earlier quoted context omitted.

That was the first thing I thought of when I saw the image. That's a very expensive computer that you risk destroying when the 300lb guy in front of you decides to lean back. The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex trying to make your arms as small as possible to tap on the keys. And the vertical viewing angle to your screen sometimes prevent…

Obese people (250lb+) shouldn't even be allowed in Economy.

Man, I hate being rude because I myself weighed 230lbs once upon a time, I get it. I just dealt with a 200lb+ man who spread his legs past the arm rests. Pissed me off the whole flight because I had to contort my body in my own seat so he wasn't spilling into me.

Re: Running local LLMs offline on a ten-hour flight

#67

Post by garethsprice »

JSR_FDED wrote: post #38
stavros wrote: post #36

Earlier quoted context omitted.

I got some Xreal glasses and it's made flights so much more enjoyable. I can watch movies or work on something lying back, and the "screen" looks massive.

I’ve been so tempted but some of the reviews say it’s not good for reading code. What’s been your experience? What is the effective resolution of the screen you get? Is it sharp enough for coding?

Currently working on an Amtrak with XReal One Pro glasses and a ThinkPad bluetooth keyboard from my Macbook Pro that is folded up in the seat pocket.

They are "OK enough" that it will be a matter of taste if they are acceptable or not at this point for you to use.

For coding they work fine for me, terminal tools work particularly well as I can bump the font size up. IDEs and web browsing aren't bad either, it's about the equivalent of a single 1080p screen. They are nicer than hunching over a laptop for travel use but I still prefer a proper monitor when available.

The optics are a generation or two from being where they need to be to market these as productivity devices, but if you like being an early adopter with all the quirks that come with it, they're fun.

Re: Running local LLMs offline on a ten-hour flight

#68

Post by 2ndorderthought »

bityard wrote: post #60
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

Hosted models are big, and there is a lot going on behind the scenes that we users have no visibility into. OpenAI, Anthropic, Google, etc do much more than just feed raw prompt tokens straight into a big 1-2TB static model and pipe the output tokens back to the web browser. The result of this is that they can do more, and end-users can get away with a lot more in terms of vague prompts and missing background. The bi…

I've done zero fine tuning in the local models I use. I also didn't do a lot of experiments except asking the 4 or 5 I downloaded what version of x package was the newest. For my work flows small models are king.

Re: Running local LLMs offline on a ten-hour flight

#69

Post by 2ndorderthought »

bachmeier wrote: post #51
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

> Sure, you can make it do something but certainly nothing useful or substantial. It works great for me. But I like to review the code and understand what it's doing, which doesn't appear to be how people do "useful or substantial" programming these days.

Everytime I am on here I am baffled by how many people just spin the wheel these days. The most important part of the sdlc for me is having humans involved in the code base. Can't plan improvements, features, refactors, etc if you don't know what the code looks like. But here we are I guess.

Re: Running local LLMs offline on a ten-hour flight

#70

Post by fy20 »

sleepyeldrazi wrote: post #61
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

I have been testing and using Qwen3.6 27B (running from my 3090) since it dropped and I genuinely think this is the first consumer hardware-grade model that can actually replace frontiers for a lot of workloads. I ran 8 tests on a variety of open-weights models, and opus 4.7 (1mil ctx version) and the little dense model was right behind it: https://github.com/sleepyeldrazi/llm_programming_tests/tree/... Of note is th…

Running it on a Macbook Pro M5 48GB:

        -hf unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL \ 
        -c 128000 \
        --parallel 1 \
        --flash-attn on \
        --no-context-shift \
        --cache-type-k q8_0 \
        --cache-type-v q8_0 \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --presence_penalty 0.0 \
        --reasoning on \
        --jinja \
        --chat-template-kwargs "{\"preserve_thinking\": true}" \
        --spec-type ngram-simple \
        --draft-max 64 \
        --timeout 1800
Maybe someone knows any tips to optimise prompt processing as that's the slowest part? It takes a few minutes before OpenCode with ~20k initial context first responds, but subsequent responses are pretty fast due to caching.
Post reply on HN