261–270 of 638 posts
Earlier quoted context omitted.
A new base model mac mini is $900. That is 45 month of Gemini. Gemini 4.7 Flash will give better OCR results that Qwen or GLM w/ 10GB.
Do you believe Gemini will costs the same in 45 months or even exist, given Google track record ?
I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have. I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me. Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardw…
Local coding is a step down but good enough for a lot of things if you have privacy concerns.
Earlier quoted context omitted.
> Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. Modal significantly improves this. Highly recommend.
Is Modal at all similar to Vast.ai or just related because "It's for AI"? I looked at Modal's page for training, and it talks about using some SDK and other junk, can you not just get a beefy instance from Modal with tons of VRAM to do what you want with?
Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
This is just so incredibly disrespectful to so many people.
Earlier quoted context omitted.
They use the datacenters for that.
No, that's for generating the hot air used to help inflate their egos. Heating is a very separate line item on the budget.
Earlier quoted context omitted.
Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.
I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience…
It is really a once in a lifetime deal.
Once the deal is over the local models will be better than what I am using now anyway and the hardware will be all the better than what I can get now for the price.
Earlier quoted context omitted.
I don't think it is really about "sensitive", but basically about any content you put in. Why would you give corporations your reasoning (data on how you interact with AI, how you "talk" etc.). All of this is private, but not necessarily sensitive. You never know what is happening with this data. They might say they don't log it or don't sell it, then few years later you'll find it all online or read a book that has…
Although in this use case, it's likely because GPT guided you to write the same story as somebody else. Talking with an LLM about an idea is a great way to make it more predictable and homogenized. If you're fixing a bike or writing software, this is usually a good thing.
Earlier quoted context omitted.
Do you believe Gemini will costs the same in 45 months or even exist, given Google track record ?
The options available across the board are getting cheaper and better all the time. There is no reason to believe that equivalent level model output will be more expensive in 12 months, let alone almost 4 years from now. Of all the good reasons to use local AI (privacy, etc), worrying about not having access to cheap models in 4 years is not one of them.
I'm not saying that local AI is necessarily economically justified right now; but it's certainly quite reasonable to think that in 4 years the subscriptions will be significantly more expensive.
I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have. I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me. Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardw…
Wild oversimplification, and benchmarks vary widely, but I've read a lot of benchmarks suggesting that Qwen3.8-27B (xhigh effort) competes with near-frontier models at a lot of coding tasks. To the best of my understanding it's not going to run very feasibly in 16GB of VRAM at usable quants however.
r/LocalLLM and r/LocalLlama are noisy, but valuable sources of anecdata if you have the time (or the tokens, hah) to comb through them. You are going to see a lot of modest setups there, and also guys with $20K+ of hardware.
The two things (besides my bank account) that keep me from investing heavily in local are (1) we are not guaranteed to get a steady release of open models in the future (2) a lot of the "fun" stuff LLM stuff that interests me involves orchestrating lots of parallel agents, which of course multiples the hardware you need to achieve it.
For example, I've been having good results having both Sol and Opus review the same PR, and then I have them cross-review each others' PRs. A next step I'd like to consider is maybe having a swarm of Luna agents review the same PR and have them fight it out... maybe with Sol doing final arbitration? I suspect 5-10 Lunas might outperform a single Opus. Or maybe not. But at any rate, that would be impractical in a homelab without a pretty big hardware (or time) budget.