Earlier quoted context omitted.
That's true if you're passing messages between identical models. There's a question to ask as to whether different models trained for different tasks would be better than single, multipurpose models though. My gut feel is that eventually multipurpose models will win because you don't have the embedded cost of relearning what syntactic structure is , but for a given training time and number of weights it's not clear w…
Could be the same base multipurpose models, but finetuned to be somewhat specialized.
Yeah, same principle. If you're passing messages between things that will react exactly the same to the same prompt, there's not a lot of point (unless the parallelism is important). If you've got fine-tunes, the whole point is that they will be better at some questions than the baseline.
Mind you, there's another idea there about mixture of experts as implemented by deciding which fine-tune to load, depending on the prompt itself... I'm sure that's been looked at.