Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

patrickmccanna.net

Post reply on HN
Print view View original

11–20 of 76 posts

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#11

Post by SyneRyder »

0o_MrPatrick_o0 wrote: post #3
SyneRyder wrote: post #2

TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.

Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.

Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.

But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#12

Post by embedding-shape »

andai wrote: post #8
DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)

Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#14

Post by cyanydeez »

embedding-shape wrote: post #12
andai wrote: post #8

Earlier quoted context omitted.

The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)

Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.

the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1.

A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do.

I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that.

But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness.

And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do.

And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#16

Post by birdsongs »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out.

Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.

How do you all keep your context so low? Complex tasks just balloon it in my experience.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#17

Post by vardalab »

0o_MrPatrick_o0 wrote: post #3
SyneRyder wrote: post #2

TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.

Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.

Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.

These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#18

Post by Octoth0rpe »

stackedinserter wrote: post #9

The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.

I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#19

Post by gchamonlive »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

> your prompt is confusing, unfocused, and doesn't work right on any LLM

You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc...

I could see this scale to multiple kiltobytes of metadata in the prompt easily.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#20

Post by hedgehog »

stackedinserter wrote: post #9

The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.

This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.
Post reply on HN