The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.
Not anymore. Qwen 3.8 27B is so great that 5k investment gives you excellent speed and quality.
Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
21–30 of 75 posts
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#22Earlier quoted context omitted.
Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on…
3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#23Earlier quoted context omitted.
the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on…
3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
this is what I use: https://github.com/Opencode-DCP/opencode-dynamic-context-pru... ; this and openviking appear to be all I need to get a proper 'do anything' setup.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#24The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.…
> How do you all keep your context so low? Complex tasks just balloon it in my experience.
By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#25The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.…
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases.
To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).
To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.
To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.
When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#26Why are you using Ollama?
Just use llama.cpp
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#27TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Your hardware can do way more than 64k tokens context window, can't it?
And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#28Earlier quoted context omitted.
3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
this is what I use: https://github.com/Opencode-DCP/opencode-dynamic-context-pru... ; this and openviking appear to be all I need to get a proper 'do anything' setup.
Thanks, I hadn't seen OpenViking before. I do everything in Claude Code right now due to the cost advantage but between the advances in local models (especially Qwen family) and models like Gemini 3.8 Flash it may be worth revisiting.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#29Earlier quoted context omitted.
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.…
> How do you all keep your context so low? Complex tasks just balloon it in my experience. By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#30The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
> "until the context rot and sampling problem is fixed forever"
I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)