401–410 of 421 posts
Interesting for my 8GB VRAM system, but the system RAM requirement seems to balloon quickly, and it starts misspelling words. Also token/s drops off quickly it seems
Earlier quoted context omitted.
Nope, lol. Large models still are quite far ahead, don't be fooled that even Gemma:31b (which is better than the 12b overall) is anywhere close to big models. There is definitely room for optimization, but fundamentally, for complex tasks, you need visible small gradients for accuracy that allow the model to be trained on (and consequently be followed during inference). For example, if you specify in instructions not…
It doesn't matter if Large models are undeniably better, if a local model is "good enough" to handle the task. With API costs ramping up, I think a lot of companies are going to want to look into what can be run locally instead, possibly only using larger models when the local models fall short.
https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense
should run perfect for 12-16gb with maybe 10-20k context
seems intelligent enough that I would recommend this as a daily driver for friends who just want a local ai that can do most things relatively quickly (getting 10 tps on my m2 air)
Earlier quoted context omitted.
Thank you for sharing your usecase! I like your product very much! Could you talk a bit how you did the finetuning? Did you use unsloth or any other tool and how went the verification to proof the outcome?
Thank you! Yea absolutely, but man, where to even start, it is very specific. Fundementally I didn't use any wrappers like unsloth or axolotl, although I have used the latter before a year or two back and it was good, but I needed something very very custom. I also wanted the whole fine tuning pipeline to exported OpenVino model to be seamless. I heavily leaned on codex, claude and some manual sleuthing around the in…
Earlier quoted context omitted.
It was almost certainly not trained for coding, as it's got both audio and vision input, is only 12B, and nowhere in the announcement is coding mentioned. It will likely not have good performance on coding in general, compared to other small models like Qwen 3.6 35B A3B, Gemma 4 26B A4B, Nvidia Nemotron 3 Nano 30B-A3B, gpt-oss-20b. For 16GB laptops, Qwen 3.5 9B is the undisputed champ. Gemma 4 31B is the top dog at s…
Qwen 3.5 9B is great for coding, but somehow, based on a few hours of subjetive tests, the Gemma 4 12B seems even better.
I had horrible luck with Gemma 4 12B with a variety of coding harnesses - but as usual Qwen 3.5 9B did OK.
EDIT: CORRECTION: I pulled a fresh copy of Gemma 4 12B and inference code and the tool use problems in my test harnesses are fixed. Gemma 4 12B is slow on my 16B MacBook Air, put produces OK results.
Earlier quoted context omitted.
I would contend that the actual big story is the gallery app: https://developers.google.com/edge/gallery Anyone with a 16GB Mac — that is quite a lot of journalists, surely — can download that, install a model into it, and play. Surely journalists have to start asking questions at least about OpenAI's consumer revenue projections now. I am a major, major AI cynic, but I decided to be an informed cynic so I've been pl…
I had discounted Edge Gallery because it didn't support system prompts, but now it does so I will give it another go. I believe the implementation does use MTP since I got an update to Gemma-4-E4B on iOS indicating such, and on macOS it's very speedy. However, on my 18GB RAM MacBook Pro, selecting Gemma-4-12B-it results in this error: > The model "Gemma-4-12B-it' requires more memory (RAM) than is available on your d…
https://blog.google/innovation-and-ai/technology/developers-...
Looks like the 12B model should fit now?
Earlier quoted context omitted.
I find ram crazy. My thinkpad has 32G of ram, it's a t470 that's nearly a decade old Why do people with modern laptops have such little amounts of ram?
Unified memory is soldered to the motherboard and needs to be ordered with the new laptop, for prices that are well above what the equivalent amount of SODIMM would cost. Fine if work's paying, but for personal devices (that might have been purchased before local models got good), people have what they have.
Earlier quoted context omitted.
/r/localllama for a while now seems to prefer Gemma 4 E4B for creative writing (especially the uncensored GGUFs).
Do they prefer E4B over the larger models or is it a matter of what fits their machine? I assume 4B isn't large enough to get interesting writing but I don't know anything about it.
Creative writing is not my focus, so this is only secondhand information. r/LocalLlama tends to focus more on the technical side; if you want more creative side check out r/SillyTavern as well (but type of info does bleed over between them).
Earlier quoted context omitted.
I had discounted Edge Gallery because it didn't support system prompts, but now it does so I will give it another go. I believe the implementation does use MTP since I got an update to Gemma-4-E4B on iOS indicating such, and on macOS it's very speedy. However, on my 18GB RAM MacBook Pro, selecting Gemma-4-12B-it results in this error: > The model "Gemma-4-12B-it' requires more memory (RAM) than is available on your d…
I wonder if they were just slightly ahead of this announcement? https://blog.google/innovation-and-ai/technology/developers-... Looks like the 12B model should fit now?