Earlier quoted context omitted.
Speak for yourself, I prefer generalist LLMs. Also, the bitter lesson of ML applies.
> I prefer generalist LLMs. No you don't. You personally probably don't need a Pokemon encyclopedia or Bengali spell checking in your daily LLM usage. These are the kinds of things you're paying for when you're using the newer models with huge parameter counts.
while I probably won't need pokemon knowledge from an llm, I still think broad pre-training enables unpredictable cross-domain knowledge transfer - seemingly irrelevant data can contribute to reasoning abilities that improve specialized tasks. and most developers work across domains anyway, so better to start general and post-train than lose emergent capabilities from narrow foundations.
even within 'coding' there's huge breadth. you might need to understand database schemas, api documentation, domain-specific algorithms, legacy system constraints, or business logic from completley different industries. a narrowly-tuned model might excel at leetcode problems but struggle when you need to build a financial trading system or whatever.