Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local LLMs offline on a ten-hour flight

deploy.live

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Running local LLMs offline on a ten-hour flight

#51

Post by bachmeier »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

> Sure, you can make it do something but certainly nothing useful or substantial.

It works great for me. But I like to review the code and understand what it's doing, which doesn't appear to be how people do "useful or substantial" programming these days.

Re: Running local LLMs offline on a ten-hour flight

#54

Post by Der_Einzige »

ryandrake wrote: post #20
rootusrootus wrote: post #7

Earlier quoted context omitted.

I'd probably choose the window seat myself, because while it is cramped, it is predictably so. When I sit in an aisle seat, it's not as cramped but I regularly get shoulder checked by passing people or beverage carts. What really makes me nervous if I'm in an economy seat is the seat in front of me. Depending on how the seat is designed, if the person suddenly reclines (or hell, just flexes the seat a bunch while mov…

That was the first thing I thought of when I saw the image. That's a very expensive computer that you risk destroying when the 300lb guy in front of you decides to lean back. The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex trying to make your arms as small as possible to tap on the keys. And the vertical viewing angle to your screen sometimes prevent…

Obese people (250lb+) shouldn't even be allowed in Economy.

Re: Running local LLMs offline on a ten-hour flight

#55

Post by Der_Einzige »

proxysna wrote: post #9
deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

You need to set sampling parameters for the llm. Had the same issue with Qwen3.5 when i first started. You can grab them off the model card page usually. From Qwen3.6 page: Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalt…

min_p author here. min_p is strictly better than top_p and top_k. The big labs don't know shit about sampling, and give absolutely nuts recommendations like this.

set min_p to like 0.3 and ignore top_p and top_k and you'll be fine.

There's better samplers now like top N sigma, top-h, P-less decoding, etc, but they're often not available in your LLM inference engine (i.e. vLLM)

Re: Running local LLMs offline on a ten-hour flight

#56

Post by RALaBarge »

mft_ wrote: post #50
2ndorderthought wrote: post #26

Earlier quoted context omitted.

Qwen3.6 is brand new. But also, search engines are so plastered with AI slop that is written by tools and companies that have no interest in you using local models. Ollama makes it 1 command to run local small models, but with the newest ones there can be kinks to work out first. /R/localllama is okay for some information but beyond that there is so much noise and very little signal. I think it's intentional.

Thanks. I’ve been experimenting with local models for over a year now, on and off, so this isn’t just limited to the latest Qwen. Anyway, I have no problem running them, but there’s a huge difference between running something via a chat interface and running it a la Claude Code so that it can interact with the local environment and create/edit files. This is the aspect that’s difficult, in my experience.

It’s all about tooling, if the ai can fetch data it can do something rad with it. Use something like an ai harness to have an mcp server and other tooling to improve the harness and the tools I made this for my own learning: GitHub.com/ralabarge/beigebox

Re: Running local LLMs offline on a ten-hour flight

#57

Post by zrail »

Der_Einzige wrote: post #54
ryandrake wrote: post #20

Earlier quoted context omitted.

That was the first thing I thought of when I saw the image. That's a very expensive computer that you risk destroying when the 300lb guy in front of you decides to lean back. The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex trying to make your arms as small as possible to tap on the keys. And the vertical viewing angle to your screen sometimes prevent…

Obese people (250lb+) shouldn't even be allowed in Economy.

Sweet, free upgrades!

Edit to be slightly less obtuse: surely you're not implying that a common carrier be allowed to discriminate based on facts about a passenger's body without making reasonable accommodations. Surely you're not implying that obese people not be allowed to fly at all. Surely you're not suggesting that fat people should just remove themselves from society so you don't have to deal with them.

Therefore, obese people should get free upgrades to economy plus or better. Thanks for the idea!

Re: Running local LLMs offline on a ten-hour flight

#59

Post by hansvm »

Der_Einzige wrote: post #54
ryandrake wrote: post #20

Earlier quoted context omitted.

That was the first thing I thought of when I saw the image. That's a very expensive computer that you risk destroying when the 300lb guy in front of you decides to lean back. The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex trying to make your arms as small as possible to tap on the keys. And the vertical viewing angle to your screen sometimes prevent…

Obese people (250lb+) shouldn't even be allowed in Economy.

Maybe.

On the other hand, in economy on some planes I just literally don't fit in a forward direction because of femur length and cycling muscles, I don't fit in a sideways direction either because of broad shoulders and arm muscles, and I don't comfortably fit vertically on some planes with fixed-position headrests which push into the middle of my shoulder blades and have me hunched the whole flight.

I'm also not _that_ big. I'm 6'2" and have lived my life moderately actively. That's it. I'm biased, but I believe economy should be designed so that I can fit too.

If you agree with that premise, that'd leave plenty of space for most 250lb people too, and there'd be no reason to exclude them.

Re: Running local LLMs offline on a ten-hour flight

#60

Post by bityard »

deanc wrote: post #4

This has been exactly my experience too. I've tried multiple harnesses (pi, claude code, codex) with multiple variants of qwen3.6 and gemma4 driven by both o mlx and ollama - and every single time I try to do anything meaningful I end up in a loop. On a 64GB Macbook Pro M3 Max. I really don't know what the hell people are doing locally, and suspect a lot of the hype around running these models locally is bullshit. Su…

Hosted models are big, and there is a lot going on behind the scenes that we users have no visibility into. OpenAI, Anthropic, Google, etc do much more than just feed raw prompt tokens straight into a big 1-2TB static model and pipe the output tokens back to the web browser. The result of this is that they can do more, and end-users can get away with a lot more in terms of vague prompts and missing background.

The biggest lesson I've learned working with local models so far is: with the smaller models, you have to understand their limitations, be willing to run experiments, and fine-tune the heck out of everything. There are endless choices to be made: which model to use, which quant, thinking or not, sampling parameters, llama.cpp vs vLLM, etc. They much more fiddly for serious work than just downloading Claude Code and having it one-shot your application. But some of us enjoy fiddling so it all works out in the end.

Post reply on HN