Useful tip. From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ). With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sen…
I think control should be top of the list here. You're talking about building work flows, products and long term practices around something that's inherently non-deterministic. And the probability that any given model you use today is the same as what you use tomorrow is doubly doubtful: 1. The model itself will change as they try to improve the cost-per-test improves. This will necessarily make your expectations non…
The rest of your points are why I mentioned AI evals and regressions. I share your sentiment. I've pitched it in the past as "We can’t compare what we can’t measure" and "Can I trust this to run on its own?" and how automation requires intent and understanding your risk profile. None of this is new for anyone who has designed software with sufficient impact in the past, of course.
Since you're interested in combating non-determinism, I wonder if you've reached the same conclusion of reducing the spaces where it can occur and compound making the "LLM" parts as minimal as possible between solid deterministic and well-tested building blocks (e.g. https://alexhans.github.io/posts/series/evals/error-compound... ).