Self-hosted Ollama is the best.
Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
31–40 of 75 posts
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#32The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.…
My guess is your sessions are interactive conversations rather than planning a course of action then, separately, executing each piece.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#33The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
Attention is all you need.
And there's only so much attention to go around.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#34[flagged]
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#35The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
> your prompt is confusing, unfocused, and doesn't work right on any LLM You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc... I could see this scale to multiple kiltobytes of metadata in the prompt easily.
That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs.
RAG was supposed to be the way out on that, and ended up being mostly abandoned.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#36[flagged]
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#37The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
> "until the context rot and sampling problem is fixed forever" I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
A lot of this is managed by the inference engine, and has nothing to do with the model.
Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).
If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#38Earlier quoted context omitted.
> your prompt is confusing, unfocused, and doesn't work right on any LLM You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc... I could see this scale to multiple kiltobytes of metadata in the prompt easily.
That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs. RAG was supposed to be the way out on that, and ended up being mostly abandoned.
That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#39I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#40Earlier quoted context omitted.
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.…
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where…
I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.