Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local LLMs offline on a ten-hour flight

deploy.live

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Running local LLMs offline on a ten-hour flight

#81

Post by garethsprice »

walrus01 wrote: post #37

As much as it's a fun gimmick to run a relatively good sized LLM like qwen 3.6 35B locally, I would much rather have the ability to run it remotely on a piece of hardware I control via VPN session. Much better on battery life and heat. If I'm on an airplane I care about having as much battery life as possible. Let's say you have a basic setup like llama.cpp and llama-server on a remote server (even if it's just sitti…

This is what I am doing - it is rare that I'm in a situation with no Internet while traveling, but very often there is an intermittent connection. Using local models or even hosted foundation models is frustrating exercise in cancelled jobs and timeouts, but Tailscale + mosh + tmux is a really nice way to connect to a workstation and resume from where the session left off - or leave it running doing its thing and come back to it later.

Same with running my local dev environment's docker containers, now they run on that workstation and my battery life is far higher, treating my portable device as a dumb terminal.

Re: Running local LLMs offline on a ten-hour flight

#84

Post by mtrifonov »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

[flagged]

Re: Running local LLMs offline on a ten-hour flight

#85

Post by JSR_FDED »

Der_Einzige wrote: post #55
proxysna wrote: post #9

Earlier quoted context omitted.

You need to set sampling parameters for the llm. Had the same issue with Qwen3.5 when i first started. You can grab them off the model card page usually. From Qwen3.6 page: Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalt…

min_p author here. min_p is strictly better than top_p and top_k. The big labs don't know shit about sampling, and give absolutely nuts recommendations like this. set min_p to like 0.3 and ignore top_p and top_k and you'll be fine. There's better samplers now like top N sigma, top-h, P-less decoding, etc, but they're often not available in your LLM inference engine (i.e. vLLM)

I’m wondering though, what does extra creativity in code generation actually look like? How is the creativity expressed in code? Does the LLM reach for Bubble Sort instead of Quicksort? Maybe it decides that sorting only the first 10 elements of an array is enough? Funny variable names? Cursing in comments?

Re: Running local LLMs offline on a ten-hour flight

#86

Post by phantomathkg »

Der_Einzige wrote: post #54
ryandrake wrote: post #20

Earlier quoted context omitted.

That was the first thing I thought of when I saw the image. That's a very expensive computer that you risk destroying when the 300lb guy in front of you decides to lean back. The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex trying to make your arms as small as possible to tap on the keys. And the vertical viewing angle to your screen sometimes prevent…

Obese people (250lb+) shouldn't even be allowed in Economy.

How about having a law that ban super tight economic seat?

Re: Running local LLMs offline on a ten-hour flight

#87

Post by rgovostes »

mtrifonov wrote: post #84
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

[flagged]

And was this written by a local model or a frontier cloud one?

Re: Running local LLMs offline on a ten-hour flight

#88

Post by datadrivenangel »

walrus01 wrote: post #37

As much as it's a fun gimmick to run a relatively good sized LLM like qwen 3.6 35B locally, I would much rather have the ability to run it remotely on a piece of hardware I control via VPN session. Much better on battery life and heat. If I'm on an airplane I care about having as much battery life as possible. Let's say you have a basic setup like llama.cpp and llama-server on a remote server (even if it's just sitti…

Agreed. I got a beefy M5 MBP for local llms and for sustained inference it gets hot enough that I worry it may end up shortening its life.

Re: Running local LLMs offline on a ten-hour flight

#89

Post by cadamsdotcom »

sleepyeldrazi wrote: post #61
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

I have been testing and using Qwen3.6 27B (running from my 3090) since it dropped and I genuinely think this is the first consumer hardware-grade model that can actually replace frontiers for a lot of workloads. I ran 8 tests on a variety of open-weights models, and opus 4.7 (1mil ctx version) and the little dense model was right behind it: https://github.com/sleepyeldrazi/llm_programming_tests/tree/... Of note is th…

> I have been testing

With local models which are often benchmaxxed, testing unfortunately isn’t as predictive as you’d like.

Re: Running local LLMs offline on a ten-hour flight

#90

Post by ntcho »

davidcox143 wrote: post #64
cube00 wrote: post #63

MacBook cable: 94W delivered Return flight will test this with the correct cable. I expect at least 16% improvement against the 70W cap Some plane sockets cut out completely if you attempt to draw more then the limit rather then continuing to provide power at the limit.

The MBP also throttles surprisingly easily. I have a 16 MBP and ended up buying a cooling stand that uses a 20w peltier cooler (solid state heat pump). It’s fixed the throttling completely, although I’m somewhat nervous about forgetting to turn the cooler off and creating condensation inside the case…

Have you tried MacsFanControl? MBP fans rarely turn on by themselves, but controlling yourself makes throttling not a problem
Post reply on HN