I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
Isn't that more or less how GPT-4 works? multiple "expert" LLMs giving input depending on the context?[0] [0] https://the-decoder.com/gpt-4-architecture-datasets-costs-an... the biggest issue is if you have too many specialists and spin a lot of them to reply to the same query and after that discard the less optimal answers. Your answer quality might improve, but the computing costs could skyrocket without some smart…
Basically the idea is that there's some pars of the model (attention/embedding) that should be trained on everything and used in every inference and other parts (the FFNN) that are fine to specialize on certain types of data (via a routing module that is also trained).
[0] https://arxiv.org/pdf/1701.06538.pdf [1] https://arxiv.org/pdf/2112.06905.pdf
EDIT: Specifically GLaM model architecture. Each MoE layer (the bottom block) is interleaved with a Transformer layer (the upper block). For each input token, e.g., ‘roses’, the Gating module dynamically selects two most relevant experts out of 64, which is represented by the blue grid in the MoE layer. The weighted average of the outputs from these two experts will then be passed to the upper Transformer layer. For the next token in the input sequence, two different experts will be selected.