Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

patrickmccanna.net

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#61

Post by 0o_MrPatrick_o0 »

vardalab wrote: post #17
0o_MrPatrick_o0 wrote: post #3

Earlier quoted context omitted.

Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.

Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par…

Thank you. This is the hackernews I miss.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#62

Post by 0o_MrPatrick_o0 »

hermitShell wrote: post #27
0o_MrPatrick_o0 wrote: post #3

Earlier quoted context omitted.

Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.

Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider. I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now. So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doin…

Yes.

I have to observe:

Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.

Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.

I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#63

Post by hedgehog »

bitexploder wrote: post #56
hedgehog wrote: post #20

Earlier quoted context omitted.

This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.

Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done. Comes down to how much of the ambiguity we expect out of the model.

What would you build? Flash Next on 128GB works ok, 3.8 27B might technicially work on less but it's just too slow at least on current APU style chips. I'm curious about the non-NVIDIA 32GB discrete GPU options but I haven't tried yet.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#64

Post by persedes »

DiabloD3 wrote: post #35
gchamonlive wrote: post #19

Earlier quoted context omitted.

> your prompt is confusing, unfocused, and doesn't work right on any LLM You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc... I could see this scale to multiple kiltobytes of metadata in the prompt easily.

That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs. RAG was supposed to be the way out on that, and ended up being mostly abandoned.

RAG was supposed to be the way out on that, and ended up being mostly abandoned.

Unless you're a contracting agency!

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#66

Post by chrisweekly »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

The "dumb zone" threshold is fuzzy but comes waay before 250k tokens. Like half that.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#67

Post by bitexploder »

hedgehog wrote: post #63
bitexploder wrote: post #56

Earlier quoted context omitted.

Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done. Comes down to how much of the ambiguity we expect out of the model.

What would you build? Flash Next on 128GB works ok, 3.8 27B might technicially work on less but it's just too slow at least on current APU style chips. I'm curious about the non-NVIDIA 32GB discrete GPU options but I haven't tried yet.

Flash next only needs 64GB for core model inference. If you really wanted it.

Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm.

Maybe older GPUs work.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#68

Post by fghorow »

bitexploder wrote: post #55
fghorow wrote: post #39

I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills. I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do? (I kind of hate to abandon Claude Code, as it seems t…

OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.

Thanks. I'll kick the tires on OMP a little harder...

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#69

Post by [deleted] »

0o_MrPatrick_o0 wrote: post #62
hermitShell wrote: post #27

Earlier quoted context omitted.

Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider. I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now. So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doin…

Yes. I have to observe: Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools. Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though. I settled on 64k toke…

[deleted]

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#70

Post by prettyblocks »

Havoc wrote: post #65

Can a 27b model even do meaningful security tasks? I thought the interesting cyber stuff is really at the edge of frontier

It depends what you mean by meaningful, but yes, qwen3.8 27b is pretty mind blowing to me. It can easily solve portswigger labs for example at q4_k_m.
Post reply on HN