I have a Django app with 54 LLM-backed workflows. Up until recently I've exclusively used Anthropic models via AWS Bedrock but just set up OpenRouter to test the new Gemini models given they seem to match Sonnet/Haiku intelligence but with 3-5x output speed. I'm using Pydantic AI for validation/normalization.
I currently maintain a registry that describes a workflow's purpose, what we're optimizing for (intelligence, speed, cost), its eval, and a human-readable bar that must be achieved.
I'm curious what strategies/systems people are using to keep track of everything and ensure an optimal model is being used for a given workflow.