Earlier quoted context omitted.
That’s like saying cloud computing is spiritually incompatible with tinkering.
It is. You can't tinker with someone else's machine.
You can’t tinker with someone else’s machine but you can tinker through it.
Local AI needs to be the norm
631–640 of 804 posts
Re: Local AI needs to be the norm
#632How is having local AI going to produce a result that's any better than using OpenAI or Anthropic? Isn't what we really need programmers who rely on themselves more than AI so they avoid technical debt accumulation?
Re: Local AI needs to be the norm
#633I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…
A phone makes a very crappy AI inference rig. It's battery powered and can't even really run at 100% utilization on an ongoing basis due to how challenging the thermals are.
Re: Local AI needs to be the norm
#634I'm literally working on an iOS app right now that needs to infer some input fields from free text typed by the user. Now to take into consideration typos, unstructured text (pricing, dates .. etc), I was pondering a cloud LLM or a basic local parser or even a local on-device LLM (ANE for 15+ devices and a different on-device LLM for the older models) For the different on-device LLM, I literally went to HuggingFace a…
[deleted]
Re: Local AI needs to be the norm
#635I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…
A phone makes a very crappy AI inference rig. It's battery powered and can't even really run at 100% utilization on an ongoing basis due to how challenging the thermals are.
at the moment yes. The one possible silver lining with all of the current hardware crunch is that it _should_ force some hardware advancements. The last couple years hardware has been kinda boring. My m1max is still zippy as all hell and doesn't really need to be upgraded, unless I am committing to local AI inference.
Re: Local AI needs to be the norm
#636Earlier quoted context omitted.
Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?
But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.
Re: Local AI needs to be the norm
#637[flagged]
Re: Local AI needs to be the norm
#638Earlier quoted context omitted.
> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…
Can you share how you use it to categorize trip photos!
this is one of the most popular options. Self hosted.
https://immich.app/
Re: Local AI needs to be the norm
#639Earlier quoted context omitted.
> They don't want you to have access to frontier models. And you will not have it. See Mythos as Exibit A. "They" fully well know that they current frontier model are maybe 6 month ahead of what people will have access to without their control. See Deepseek as Exibit B The reason you can't run these locally are more with the fact that those mythos sized models require extreme amount of memory and processing power to…
Although, I wonder how many orders of magnitude in terms of affordability the utilization rate actually gets them. Realistically if you use a self-hosted LLM for your job, you might be using it, what, a solid 6 hours per day? Assuming you can keep it actually fed, while working (so, some agentic thing might be necessary, I guess it will need to be more than VSCode autocomplete and responding to individual prompts). A…
> Realistically if you use a self-hosted LLM for your job, you might be using it, what, a solid 6 hours per day?
You can grow the utilization rate well beyond that if you don't always care about getting a quick, real-time response. (And if you do, then maybe the cloud model was the better deal after all!)
Re: Local AI needs to be the norm
#640(I am not an expert on anything.) One happy circumstance here is that while the RAM cartel is chasing Big AI's money today, in the medium term its self-interest probably makes it a supporter of local AI. A new, compelling reason to have 128GiB, 256GiB or more of VRAM on all your devices? You can be sure that the dollar signs are glowing in their eyes already. The less efficient use of VRAM by personal devies (any giv…
Just because we'll have to pay for the hardware, doesn't mean we'll have meaningful control. Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption, yet costing more than a vastly superior laptop; quasi-mandatory platforms for highly addictive experiences, centered around the flow of information.
And now with LLMs we can create even more fabulously addictive experiences, even more finely tuned information flows, even more treacherous servants. I very much doubt that we'll be allowed full control of it all. Every effort will be spent to centralize power, and every effort will be spent to extract as much cash as possible from us for the privilege.