71–80 of 804 posts
Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference. Damned if they do, damned if they don't.
Earlier quoted context omitted.
You're maybe missing the article's point, which is to use local models appropriately: > “But Local Models Aren’t As Smart” > Correct. > But also so what? > Most app features don’t need a model that can write Shakespeare, explain quantum mechanics, and pass the bar exam. They need a model that can do one of these reliably: summarize, classify, extract, rewrite, or normalize. > And for those tasks, local models can be…
This is a bit naive IMHO... I have tried quite a bunch of local models, and the reality is that it's not just a matter of of "it's a small model that should be hostable easily". Its also a matter of whats your acceptable prefill TTFT and decode t/s. All the local models I used, on a _consumer grade_ server (32GB DDR5, AMD Ryzen) have been mostly unusable interactively (no use as coding agent decently possible), and e…
2) It's probably not the time/place to trouble-shoot your "consumer grade server" LLM experience, but if you're running on CPU (you don't mention a GPU) then yeah, your inference speed will be slow.
3) Counterpoint: my consumer-grade Macbook Pro (M1 Max, 64GB) runs Qwen3.6-35B-A3B fast enough to be very usable for regular interactive coding support. (And it would fly with smaller models performing simpler tasks.)
Question: for software development, how much of an AI do you need for local development? Can it be run locally? Can someone train something that knows a lot about software but lacks comprehensive coverage of history, politics, and popular culture?
Earlier quoted context omitted.
If it was such a good and laudable idea why didn't they tell me about it before they activated it? It seems to me like they avoided it in the hopes that I wouldn't notice, because, presumably if I had, I would have IMMEDIATELY disabled it. Also why doesn't their task manager show that it's actually the one downloading? Why does it go out of it's way to hide this activity? Since I have conky on my desktop I could catc…
I'm guessing you immediately close the What's New Chrome tab when you update? https://developer.chrome.com/blog/new-in-chrome-148#prompt-a... https://www.google.com/chrome/ai-innovations/ They have absolutely not been shy about any of this.
Please show me where in either of those documents it explains it's going to download a 4GB model.
Earlier quoted context omitted.
It should be feasible to crowd fund training runs right?
A training run costs somewhere in the neighborhood of a billion dollars. That’s a thousand millions. How many crowdfunded projects do you know that have raised even one percent of that? Who’s going to be in charge of collecting that scale of money? Perhaps some sort of company formed for the benefit of humanity, which will promise to be a non-profit? Some sort of “Open” AI? Oh, wait.
But, and it is not a small but, is the starting point really zero?
Question: for software development, how much of an AI do you need for local development? Can it be run locally? Can someone train something that knows a lot about software but lacks comprehensive coverage of history, politics, and popular culture?
Two issues - 1. Local models are likely to be more power-expensive to run (per-"unit-of-intelligence") than remote models, due to datacenter economies of scale. People do not like to engage with this point, but if you have environmental concerns about AI, this is a pretty important one. 2. Using dumb models for simple tasks seems like a good idea, but it ends up being pretty clear pretty quick that you just want the…