I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…
Just imagine the next PlayStation or XBox shipping with these models baked in for developer use. The kinds of things that could unlock.
Good point. Take the state of the world and craft npc dialogue for instance.
The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…
I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…
If you have capable hardware and kids, a local LLM is great. A simple system prompt customisation (e.g. ‘all responses should be written as if talking to a 10 year old’) and knowing that everything is private goes a long way for me at least.
It's striking how much of the AI conversation focuses on new use cases, while overlooking one of the most serious non-financial costs: privacy. I try to be mindful of what I share with ChatGPT, but even then, asking it to describe my family produced a response that was unsettling in its accuracy and depth. Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to…
> I try to be mindful of what I share with ChatGPT, but even then, asking it to describe my family produced a response that was unsettling in its accuracy and depth. > Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to persist. Maybe I'm missing something, but why wouldn't that be expected? The chat history isn't their only source of information - these mod…
I think you are underestimating how notable a person needs to be for their information to be baked into a model.
How up to date are you on current open weights models? After playing around with it for a few hours I find it to be nowhere near as good as Qwen3-30B-A3B. The world knowledge is severely lacking in particular.
Agree. Concrete example: "What was the Japanese codeword for Midway Island in WWII?" Answer on Wikipedia: https://en.wikipedia.org/wiki/Battle_of_Midway#U.S._code-bre... dolphin3.0-llama3.1-8b Q4_K_S [4.69 GB on disk]: correct in deepseek-r1-0528-qwen3-8b Q6_K [6.73 GB]: correct in 10 seconds gpt-oss-20b MXFP4 [12.11 GB] low reasoning: wrong after 6 seconds gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 m…
Am I the only one who thinks taking a huge model trained on the entire internet and fine tuning it is a complete waste? How is your small bit of data going to affect it in the least?
Yep, it's almost as bad as all the cars' cooling systems using up so much water.
Estimated 1.5 billion vehicles in use across the world. Generous assumptions: a) they're all IC engines requiring 16 liters of water each. b) they are changing that water out once a year That gives 24m cubic meters annual water usage. Estimated ai usage in 2024: 560m cubic meters. Projected water usage from AI in 2027: 4bn cubic meters at the low end.
what does water usage mean? is that 4bn cubic meters of water permanently out of circulation somehow? is the water corrupted with chemicals or destroyed or displaced into the atmosphere to become rain?
NVIDIA will probably give us nice, coding-focused fine-tunes of these models at some point, and those might compare more favorably against the smaller Qwen3 Coder.
What is the best local coder model that that can be used with ollama? Maybe a too opened ended question? I can run the deepseek model locally really nicely.
Probably Qwen3-Coder 30B, unless you have a titanic enough machine to handle a serious 480B model.
The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…
I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…
I am getting huge productivity gains from using models, and I mostly use them as "oracles" (though I am extremely careful with respect to how I have to handle hallucination, of course): I'd even say their true power --just like a human--comes from having an ungodly amount of knowledge , not merely intelligence. If I just wanted something intelligent, I already had humans!... but merely intelligent humans, even when g…
> I am actually able to use ChatGPT 4.5 as not just an employee, not even just as a coworker, but at times as a mentor or senior advisor: I can tell it what I am trying to do, and it helps me by applying advanced mathematical insights or suggesting things I could use. You can still do that sort of thing, but just have it perform searches whenever it has to deal with a matter of fact. Just because it's trained for too…
If you strip all the facts from a mathematician you get me... I don't need another me: I already used Google, and I already failed to find what I need. What I actually need is someone who can realize that my problem is a restatement of an existing known problem, just using words and terms or a occluded structure that don't look anything like how it was originally formulated. You very often simply can't figure that out using Google, no matter how long you sit in a tight loop trying related Google searches; but, it is the kind of thing that an LLM (or a human) excels at (as you can consider "restatement" a form of "translation" between languages), if and only if they have already seen that kind of problem. The same thing comes up with novel application of obscure technology, complex economics, or even interpretation of human history... there is a reason why people who study Classics "waste" a ton of time reading old stories rather than merely knowing the library is around the corner. What makes these AIs so amazing is thinking of them as entirely replacing Google with something closer to a god, not merely trying to wrap it with a mechanical employee whose time is ostensibly less valuable than mine.