Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

patrickmccanna.net

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#41

Post by vosper »

birdsongs wrote: post #40
gpugreg wrote: post #25

Earlier quoted context omitted.

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where…

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#42

Post by DiabloD3 »

gchamonlive wrote: post #38
DiabloD3 wrote: post #35

Earlier quoted context omitted.

That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs. RAG was supposed to be the way out on that, and ended up being mostly abandoned.

That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.

That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context.

However, this is not something that is inherently part of models or inference engine, but part of the harness.

Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#43

Post by hypfer »

birdsongs wrote: post #40
gpugreg wrote: post #25

Earlier quoted context omitted.

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where…

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails.

Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#45

Post by gpugreg »

birdsongs wrote: post #40
gpugreg wrote: post #25

Earlier quoted context omitted.

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where…

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

I stopped using GitHub Copilot extension in VS Code when they introduced their new pricing model, but I should have switched much earlier. The developers have barely a clue how LLMs work and the company structure is misaligned with creating a quality product. They do not perform benchmarks to evaluate whether new "features" are any good and instead bloat the context with more and more tools that are rarely useful and often confuse models.

Eventually, the context window got so bloated that they resorted to hiding function bodies in large files, which is of course a stupid idea because then the LLMs have to use other tools to read the files, wasting even more tokens, or hallucinate the content. Honestly, it is amazing that LLMs work at all in VS Code.

You can inspect the context by pressing F1 and then selecting "Developer: Show Chat Debug View" in VS Code (https://github.com/microsoft/vscode/wiki/Copilot-Issues) and marvel at all the garbage that is in there.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#46

Post by JamesSwift »

birdsongs wrote: post #40
gpugreg wrote: post #25

Earlier quoted context omitted.

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where…

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others context as it tries different solutions. Bonus points for having part 2 fan out on its own, though thats much harder on shared hardware.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#47

Post by birdsongs »

hypfer wrote: post #43
birdsongs wrote: post #40

Earlier quoted context omitted.

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails. Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.

I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output.

I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser.

I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me.

I spent the time it was churning practicing drawing.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#49

Post by birdsongs »

JamesSwift wrote: post #46
birdsongs wrote: post #40

Earlier quoted context omitted.

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others co…

> Sure it works fine because you havent yet experienced how much better it can be.

Better in what way, though? It solved my problem, without issue, in a very elegant way. Perhaps I could tweak and optimise to make it solve it slightly faster but honestly, I'm just not that into it. It works, idk, doesn't seem like it needs fixing to me.

But honestly, I don't care a lot about these tools, I said elsewhere I'm probably switching careers if this is the new norm, I hate this version of engineering. It's not even engineering, it's just management.

Only going all in because our company is forcing us to.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#50

Post by zozbot234 »

SyneRyder wrote: post #2

TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.

The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.
Post reply on HN