I ran Gemma 4 as a local model in Codex CLI

> Not asking, and living in ignorance, is riskier. For high-stakes questions, of course I'd want references that only an online model like ChatGPT or Gemini, etc. would be able to find. If I am asking a local model for health advice, odds are that it is because I am traveling and am temporarily offline, or am preparing off-grid infrastructure. In both cases I definitely require a best-effort answer. I also require the model to be able to tell when it doesn't know the answer.

If I was prepping, I’d want e.g. Wikipedia available offline and default to human-assisted decision-making, and definitely not rely on a 31B parameter model.

To be reductive, the ‘brain’ of any of these models is essentially a compression blob in an incomprehensible format. The bigger the delta between the input and the output model size, the lossier the compression must be.

It therefore follows (for me at least) that there’s a correlation between the risk of the question and the size of model I’d trust to answer it. And health questions are arguably some of the most sensitive - lots of input data required for a full understanding, vs. big downsides of inaccurate advice.

> If you would, ignore health advice for a moment, and switch to electrical advice. Imagine I am putting together electrical infrastructure, and the model gives me bad advice, risking electrocution and/or a serious fire. Why is electrical advice not censored, and what makes it not be high-stakes!? The logic is the same.

You’re correct that it’s possible to find other risky areas that might not be currently censored. Maybe this is deliberate (maybe the input data needed for expertise in electrical engineering is smaller?) or maybe this is just an evolving area and human health questions are an obvious first area to address?

Either way, I’m not trusting a small model with detailed health questions, detailed electrical questions, or the best way to fold a parachute for base jumping. :)

(Although, if in the future there’s a Gemma-5-Health 32B and a Gemma-5-Electricity 32B, and so on, then maybe this will change.)

4090, 128gb of ram (long before you'd have to take out a loan). I'm fairly sure it would run just as fine on a 3090.

Thanks to the settings suggestions in the article, I was able to squeeze in the 31b model. Still testing, but it's real tight in 24gb of vram. A bit slower, too, but usable. Not sure I'm seeing much of a quality boost yet, but I'm still testing.

Weird. A great number of my medical or legal queries are actually answered, but come with a disclaimer, often at the end of the inference. (I'd offer up some examples, but I'm not at the desk.)

I also find that you can coerce a wide spectrum of otherwise declined queries by editing its initial rejection into the start of an answer. For example changing the "I'm sorry I can't answer that..." response to "Here's how..." And then resubmitting the inference, allowing it to continue from there. It's not perfect, sometimes it takes multiple attempts, but it does work. At least in my experience. (This isn't Gemma-specific tip, either. Nearly every model I've tried this with tends to bend quite a bit doing this.)

It's even more strange how its not obvious to someone who uses codex extensively daily.

The rate limiting step is the LLM going down stupid rabbit holes or overthinking hard and getting decision paralysis.

The only time raw speed really matters is if you are trying to add many many lines of new code. But if you are doing that at token limiting rates you are going to be approaching the singularity of AI slop codebase in no time.

Using Gemma4-31B-q4_NL in open code with a 128k context and it’s been great.