121–130 of 286 posts
Earlier quoted context omitted.
2days :/...
Given I know people running gemma3 on local devices for over almost a month now this is either a very slow news day or evidence of finger missing the pulse... https://blog.google/technology/developers/gemma-3/
Earlier quoted context omitted.
"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
While none of that is false, I think there's a big difference from shipping your data to an external LLM API and using AWS. Using AWS is basically a "physical server they have control of".
Earlier quoted context omitted.
Given I know people running gemma3 on local devices for over almost a month now this is either a very slow news day or evidence of finger missing the pulse... https://blog.google/technology/developers/gemma-3/
This is new. These are new QAT (Quantization-Aware Training) models released by the Gemma team.
Earlier quoted context omitted.
This is new. These are new QAT (Quantization-Aware Training) models released by the Gemma team.
There's nothing more than an iteration on the topic, gemma3 was smashing local results a month ago and made no waves as it dropped...
> Last month, we launched Gemma 3, our latest generation of open models. Delivering state-of-the-art performance, Gemma 3 quickly established itself as a leading model capable of running on a single high-end GPU like the NVIDIA H100 using its native BFloat16 (BF16) precision.
> To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QAT) that dramatically reduces memory requirements while maintaining high quality.
The thing that's new, and that is clearly resonating with people, is the "To make Gemma 3 even more accessible..." bit.
I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
There is another aspect to consider, aside from privacy.
These models are trained by downloading every scrap of information from the internet, including the works of many, many authors who have never consented to that. And they for sure are not going to get a share of the profits, if there is every going to be any. If you use a cloud provider, you are basically saying that is all fine. You are happy to pay them, and make yourself dependent on their service, based on work that wasn't theirs to use.
However, if you use a local model, the authors still did not give consent, but one could argue that the company that made the model is at least giving back to the community. They don't get any money out of it, and you are not becoming dependent on their hyper capitalist service. No rent-seeking. The benefits of the work are free to use for everyone. This makes using AI a little more acceptable from a moral standpoint.
Earlier quoted context omitted.
There's nothing more than an iteration on the topic, gemma3 was smashing local results a month ago and made no waves as it dropped...
Quoting the linked story: > Last month, we launched Gemma 3, our latest generation of open models. Delivering state-of-the-art performance, Gemma 3 quickly established itself as a leading model capable of running on a single high-end GPU like the NVIDIA H100 using its native BFloat16 (BF16) precision. > To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QA…
"An iteration on a theme".
Once the network design is proven to work yes it's an impressive technical achievement, but as I've said given I've known people in multiple research institutes and companies using Gemma3 for a month mostly saying they're surprised it's not getting noticed... This is just enabling more users but the none QAT version will almost always perform better...
I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
Earlier quoted context omitted.
Quoting the linked story: > Last month, we launched Gemma 3, our latest generation of open models. Delivering state-of-the-art performance, Gemma 3 quickly established itself as a leading model capable of running on a single high-end GPU like the NVIDIA H100 using its native BFloat16 (BF16) precision. > To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QA…
As I've said in my lectures on how to perform 1bit training of QAT systems to build classifiers... "An iteration on a theme". Once the network design is proven to work yes it's an impressive technical achievement, but as I've said given I've known people in multiple research institutes and companies using Gemma3 for a month mostly saying they're surprised it's not getting noticed... This is just enabling more users b…
Earlier quoted context omitted.
As I've said in my lectures on how to perform 1bit training of QAT systems to build classifiers... "An iteration on a theme". Once the network design is proven to work yes it's an impressive technical achievement, but as I've said given I've known people in multiple research institutes and companies using Gemma3 for a month mostly saying they're surprised it's not getting noticed... This is just enabling more users b…
Sounds like you're excited to see Gemma 3 get the recognition it deserves on Hacker News then.
It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.