Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

patrickmccanna.net

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#71

Post by hedgehog »

bitexploder wrote: post #67
hedgehog wrote: post #63

Earlier quoted context omitted.

What would you build? Flash Next on 128GB works ok, 3.8 27B might technicially work on less but it's just too slow at least on current APU style chips. I'm curious about the non-NVIDIA 32GB discrete GPU options but I haven't tried yet.

Flash next only needs 64GB for core model inference. If you really wanted it. Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm. Maybe older GPUs work.

128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#72

Post by bitexploder »

hedgehog wrote: post #71
bitexploder wrote: post #67

Earlier quoted context omitted.

Flash next only needs 64GB for core model inference. If you really wanted it. Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm. Maybe older GPUs work.

128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.

Tesla V100? Would have to be a 3-ish bit quant in 64GB ram

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#74

Post by ImL1s »

fghorow wrote: post #39

I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills. I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do? (I kind of hate to abandon Claude Code, as it seems t…

[flagged]

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#76

Post by circularfoyers »

bitexploder wrote: post #57
cube00 wrote: post #13

Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385

Things are different now on a Mac. Many small improvements make Ollama genuinely decent for many models now.

The fact that those issues existed for so long while the entire time where not issues with other inference runtimes is more the point. And I think are indicative of future incompetence.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#78

Post by varsha_saini »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

[flagged]

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#79

Post by airspresso »

From what I'm seeing elsewhere, context size up to 128k should be possible on this hardware. It really matters for agentic workloads to push that context size headroom up. Anthropic are spoiling us with models that do 500k context and beyond.

Re: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

#80

Post by sbnmkatoch »

DiabloD3 wrote: post #6

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful contex…

[flagged]
Post reply on HN