31–40 of 125 posts
Earlier quoted context omitted.
The only reason why you run local models is for privacy, never for cost. Or even latency.
Autonomy generally, not just privacy. You never know what the future will bring, AI will be enshittified and so will hubs like huggingface. It’s useful to have an off grid solution that isn’t subject to VCs wanting to see their capital returned.
Kimi K2 is a really weird model, just in general. It's not nearly as smart as Opus 4.5 or 5.2-Pro or whatever, but it has a very distinct writing style and also a much more direct "interpersonal" style. As a writer of very-short-form stuff like emails, it's probably the best model available right now. As a chatbot, it's the only one that seems to really relish calling you out on mistakes or nonsense, and it doesn't h…
Earlier quoted context omitted.
I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…
As long as you're willing to wait up to an hour for your GPU to get scheduled when you do want to use it.
Earlier quoted context omitted.
I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…
I think you’re missing the whole point, which is not using cloud compute.
Earlier quoted context omitted.
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
I would assume that if the model made no assumptions, it would be unable to complete most requests given in natural language.
There are shitloads of ambiguities. Most of the problems people have with LLMs is the implicit assumptions being made.
Phrased differently, telling the model to ask questions before responding to resolve ambiguities is an extremely easy way to get a lot more success.
Earlier quoted context omitted.
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
Full instruction following looks like monkey’s paw/malicious compliance. A good way to eliminate a bug from a codebase is to delete the codebase, that type of thing. You want the model to have enough creative freedom to solve the problem otherwise you are just coding using an imprecise language spec. I know what you mean: a lot of my prompts include “never use em-dashes” but all models forget this sooner or later. Bu…
Kimi K2 is a really weird model, just in general. It's not nearly as smart as Opus 4.5 or 5.2-Pro or whatever, but it has a very distinct writing style and also a much more direct "interpersonal" style. As a writer of very-short-form stuff like emails, it's probably the best model available right now. As a chatbot, it's the only one that seems to really relish calling you out on mistakes or nonsense, and it doesn't h…
Kimi K2 is a really weird model, just in general. It's not nearly as smart as Opus 4.5 or 5.2-Pro or whatever, but it has a very distinct writing style and also a much more direct "interpersonal" style. As a writer of very-short-form stuff like emails, it's probably the best model available right now. As a chatbot, it's the only one that seems to really relish calling you out on mistakes or nonsense, and it doesn't h…
How do you feel K2 Thinking compares to Opus 4.5 and 5.2-Pro?
Earlier quoted context omitted.
The only reason why you run local models is for privacy, never for cost. Or even latency.
Autonomy generally, not just privacy. You never know what the future will bring, AI will be enshittified and so will hubs like huggingface. It’s useful to have an off grid solution that isn’t subject to VCs wanting to see their capital returned.
If anyone wants to bet that future cloud hosted AI models will get worse than they are now, I will take the opposite side of that bet.
> It’s useful to have an off grid solution that isn’t subject to VCs wanting to see their capital returned.
You can pay cloud providers for access to the same models that you can run locally, though. You don’t need a local setup even for this unlikely future scenario where all of the mainstream LLM providers simultaneously decided to make their LLMs poor quality and none of them sees this as market opportunity to provide good service.
But even if we ignore all of that and assume that all of the cloud inference everywhere becomes bad at the same time at some point in the future, you would still be better off buying your own inference hardware at that point in time. Spending the money to buy two M3 Ultras right now to prepare for an unlikely future event is illogical.
The only reason to run local LLMs is if you have privacy requirements or you want to do it as a hobby.
Earlier quoted context omitted.
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
I can't help you then. You can find a close analogue in the OSS/CIA Simple Sabotage Field Manual. [1] For that reason, I don't trust Agents (human or ai, secret or overt :-P) who don't push back. [1] https://www.cia.gov/static/5c875f3ec660e092cf893f60b4a288df/... esp. Section 5(11)(b)(14): "Apply all regulations to the last letter." - [as a form of sabotage]