441–450 of 651 posts
Have you tried optimizing for MLX? It seems like a waste to have neural cores and not use them. I've often wondered why the hype around apple neural core when 99% of software doesn't use them.
https://github.com/ml-explore/mlx-lm
Having used half the systems that Vicki mentioned, mlx was the best balance between power and ease of use. Just a pip install away.
I think this is overselling their capabilities. I've used Gemma 4 and Qwen 3.6 quite a bit on my strix halo home server. They're great models and the dense variants are significantly better, but they're still very far behind the frontier. If you boot up Gemma 4 MoE and OpenCode/Pi and expect to perform anything like Claude Code or Codex you're going to be very disappointed.
I posted this yesterday https://github.com/day50-dev/petsitter
I use it with https://github.com/day50-dev/simple-llm-cli
And modify the "tricks" until my evals get to good numbers. It's a model by model basis.
This is what the larger firms are doing - they have custom prompts per model
This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within…
The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…
Earlier quoted context omitted.
Hard to guarantee it's private if you don't keep it local... I don't have a lot of trust for companies in this space.
Yes, but I think that'll change eventually. If you trust hosting your code with a specific cloud provider then you'll probably also trust them for code assist. At least that's my theory. There'll probably need to be a threat of massive litigation should they fail to comply with such a policy.
Maybe people will trust companies, but those companies will rarely deserve that trust. Anyone that pays attention sees breach announcements almost every day. Security is never a concern for these companies until it embarrasses them. Then, as soon as the negative attention fades, security again becomes the second to last priority.
Do not trust companies with any data that is important to you unless the effective management of that data is required by law, and the laws are comprehensive.
There are good use cases for them for sure, the Gemma 4 Good hackathon a while ago showed how local models can solve problems in health and education in areas with low connectivity or small infrastructure.
Earlier quoted context omitted.
The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…
> Everyone wants to shuck the chore and the responsibility. Which gives all the power to the big techs. I'll never understand why the average company seems to have no problem with this.
I can see how it makes sense for companies, because money is "only money" but an ongoing operational distraction can be much more costly, as in, it can be detrimental to the success of the overall business.
Earlier quoted context omitted.
> DGX Spark-alike is really just asking for trouble. Prefill kills perf. You're right that prefill kills perf, but shrug the GB10 has far more compute than it has memory bandwidth, so prefill isn't it's bottleneck.
I've seen the same, Sparks are great at non time-sensitive tasks. if you can set up a agentic loop that does not require human intervention, you can design around the memory bandwidth limitations
I’m getting 40tk/s decode with 1000+tk/prefill with a 198B-A11B model on mine
Does it really needs a GPU at 300Watts to do all that tasks?
Earlier quoted context omitted.
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
You can pay, and also use deepseek-v4-flash. OpenRouter even lets you "block" or limit your usage to providers that don't train on data. Since the weights are open, other companies are already serving the model on non-DeepSeek owned hardware: https://openrouter.ai/deepseek/deepseek-v4-flash
More than that, they have various zero data retention options and provide a convenient json list of them.
Earlier quoted context omitted.
Sure, but it's also not really out of scale with the cost of a shop tool in other trades. If you're a professional that's confident in a positive return on the investment (optimal or not), or just a hobbyist with the luxury budget for a "shop" that cost is well within norms. That's not everybody, of course, but it's not some inconceivable fantasy. A lot of people in the tech community here on HN, specifically, end up…
But you can get that return from a paid service too, in fact it'll be better. So just comparing costs, what's the annualized ROI on the Mac Studio assuming it means you avoid paying $240/y for Claude? Cause I can always set aside the Mac's price in some investments and pay for Claude out of that.