191–200 of 604 posts
My max20 sub is sitting unused since april mostly now, codex with 5.4 (and now 5.5) even with fast mode (= double token costs) is night and day. Opus is doing convincing failures and either forgets half the important details or decides to do "pragmatic" (read: technical debt bandaids or worse) silently and claims success even with everything crashing and burning after the changes. and point out the errors it will mak…
Switched to local models after quality dropped off a cliff and token consumption seemed to double. Having some success with Qwen+Crush and have been more productive.
Would love some more info on how you got any local model working with Crush. Love charmbracelet but the docs are all over the place on linking into arbitrary APIs.
Edit: i forgot HN doesn't do code fences. See https://pastebin.com/2rQg0r2L
Obviously the context window settings are going to depend on what you've got set on the llama-server/llama-swap side. Multiple models on the same server like I have in the config snippet above is mostly only relevant if you're using llama-swap.
TL;DR is you need to set up a provider for your local LLM server, then set at least one model on that server, then set the large and small models that crush actually uses to respond to prompts to use that provider/model combo. Pretty straightforward but agree that their docs could be better for local LLM setups in particular.
For me, I've got llama-swap running and set up on my tailnet as a [tailscale service](https://tailscale.com/docs/features/tailscale-services) so I'm able to use my local LLMs anywhere I would use a cloud-hosted one, and I just set the provider baseurl in crush.json to my tailscale service URL and it works great.
I use Claude Code with GLM, Kimi and MiniMax models. :) I was worried about Anthropic models quality varying and about Anthropic jacking up prices. I don't think Claude Code is the best agent orchestrator and harness in existence but it's most widely supported by plugins and skills.
Where are you getting inference from? I'm overwhelmed by the options at the moment.
AI services are only minorly incentivized to reduce token usage. They want high token usage, it makes you pay more. They are going to continually test where the limit is, what is the max token usage before you get angry. All AI companies will continue to trade places for token use and cost as cost increases. We are in tepid water pretending it is a bath pretending we aren’t about to be boiled frogs.
People said this about AWS too. "Why would they save you money??". It turns out that every time they reduce prices, they make more money, because more people use their services. AI companies have the same incentive. Make it cheaper and people will use it more, making you more money (assuming your price is still above cost). And of course they have every reason to reduce their on costs.
Earlier quoted context omitted.
I have Max 5x and use only Claude Opus on xhigh mode. I don't use agents, or even MCPs, and stick to Claude Code. I find it incredibly difficult to saturate my usage. I'm ending the average week at 30-ish percentage, despite this thing doing an enormous amount of work for (with?) me. Now I will say that with pro I was constantly hitting the limit -- like comically so, and single requests would push me over 100% for t…
If only OpenAI spent a significant amount of money on some kind of generative software that was predominantly trained on internet comments that'd be able to do all the astroturfing for them...
I feel like almost everyone using AI for support systems is utterly failing at the same incredibly obvious place. The first job of any support system—both in terms of importance and chronologically—is triage. This is not a research issue and it's not an interaction issue. It's at root a classification problem and should be trained and implemented as such. There are three broad categories of interaction: cranks, grand…
Earlier quoted context omitted.
I also use it this way and I'm overall pretty happy with it, but it feels like they really want us to use it in "autopilot" mode. It's like they have two conflicting priorities of "make people use more tokens so we can bill them more" and "people are using more tokens than expected, our pricing structure is no longer sustainable" (but I guess they're not really conflicting, if the "solution" involves upgrading to a h…
autopilot (yolo mode) is amazing and feels great, truly delegate instead of hand-holding on every step
Earlier quoted context omitted.
Open Source isn't even within 50% of what the SOTA models are. Benchmarks are toys, real world use is vastly different, and that's where they seriously lag. Why should anyone waste time on poorer results? I'd rather pay my $200/mo because my time matters. I'm not a poor college student anymore, and I need more return on my time. I'm not shitting on open weights here - I want open source to win. I just don't see how t…
There's going to be a day when we look back at $200/mo price tags and say "wow that was cheap". The breakeven at this price is 6 minutes of productivity per work day for an engineer making $200k.
Are you suggesting that someone making $20k should be spending $200/mo on Claude?
Earlier quoted context omitted.
This was a real issue, and Anthropic recently awknowledged it: https://www.anthropic.com/engineering/april-23-postmortem Of course, it sucks when companies screw up ... but at the same time, they "paid everyone back" by removing limits for awhile, and (more importantly to me) they were transparent about the whole thing. I have a hard time seeing any other major AI provider being this transparent, so while I'm annoyed…
Yes that was one issue. It’s not the general degradation I have been talking about though, which is ongoing. I recall reading similar tales of woe with other providers here on HN. I think the gradual dialling back of capability as capacity becomes strained as users pile on is part of the MO of all the big AI companies.