Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

patrickmccanna.net

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#51

Post by solenoid0937 »

JamesSwift wrote: post #46
birdsongs wrote: post #40

Earlier quoted context omitted.

I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y , I say you have SSH keys to the embedded hardware: plan,…

Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others co…

Just let the model cook. All of these weird tricks hurt frontier models

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#52

Post by docjay »

DiabloD3 wrote: post #42
gchamonlive wrote: post #38

Earlier quoted context omitted.

That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.

That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context. However, this is not something that i…

You’re right about the harness being the issue. It’s really down to giving it functions specific to your use case that will let it surgically read/modify files, rather than needing to consume entire project folders. I built my own and for Python files some of the most helpful functions I provide are equivalent to:

inspect_function(filename, function, class)

replace_function()

call_graph()

And a few other convenient ones. Beyond that it’s trickery like if a function returns more than N lines I omit the result and auto-reply “Your function call was too verbose.” Typically that’s stuff like recursively listing every file in a repo to “see what it’s working with” or similar. When it emits the next call in response to it I clip the previous attempt (and my response) off the conversation and attach the new call/result as if that’s what it did in the first place. I also log that event so if the same type of thing happens often enough I’ll create a special function to address it, or modify an established one so it’s not tempted to do it again.

Language models don’t know what they know, they know what has been said. Even if you give it an entire Python environment it won’t reach for AST, but if you give it a function called Python_AST() it’ll use it every time.

I’ve never seen an off-the-shelf harness that approached it that way.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#54

Post by lowbloodsugar »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

> you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.

This was certainly true when I first tried the new models with a 1M context. After 200k things got weird pretty fast. I haven’t had that problem since Opus 4.8. I’m regularly bumping against 800k tokens in “lazy” adhoc sessions. “Lazy” in that I ought to do as you suggest, in the way that I ought to refactor this code, I ought to factor out the meat of this session, but in the moment it’s still producing useful output! Tool harness is a force multiplier too: tools that put all tool use in subagents are incredibly frugal with the main chat session.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#55

Post by bitexploder »

fghorow wrote: post #39

I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills. I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do? (I kind of hate to abandon Claude Code, as it seems t…

OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#56

Post by bitexploder »

hedgehog wrote: post #20
stackedinserter wrote: post #9

The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.

This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.

Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done.

Comes down to how much of the ambiguity we expect out of the model.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#58

Post by bitexploder »

andai wrote: post #7

> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.

20 years later… I don’t get joy from hacking things. But the 20 year wisdom is a lot of the time it doesn’t matter if it is safe. Just know when it does matter and worry about that :)

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#60

Post by 0o_MrPatrick_o0 »

dell2024 wrote: post #5

I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself. It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.

FWIW, I’m not dead ended yet.

Using frontier providers to manage prompt tuning experiments has been very exciting. Might be something there.

Post reply on HN