Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
1–10 of 75 posts
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#2TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#3TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#4TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Yes, please!
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#5I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself.
It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#6The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context.
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#7> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#8The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…
The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#9The main gotcha for local models is insane hardware requirements.
Even for $10K you get mediocre performance.
Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
#10[deleted]