How to scale LLMs better with an alternative to transformers
hazyresearch.stanford.edu
How to scale LLMs better with an alternative to transformers
1–10 of 35 posts
Re: How to scale LLMs better with an alternative to transformers
#2I wonder how a decentralized, hierarchical LLM would perform.
For example:
LLM A is trained on all of Wikipedia
LLM B is trained on all of Hacker News
LLM C is trained on all of Project Gutenberg
W sends Q to A and B.
Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?"
Would the answer be as good as or better than what an LLM which is trained on Wikipedia, Hacker News and Project Gutenberg would return?
If it is of similar quality, then we could build a hierarchical tree of consumer hardware LLMs which are hosted all over the world.
Re: How to scale LLMs better with an alternative to transformers
#3I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
[dead]
Re: How to scale LLMs better with an alternative to transformers
#4I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
Isn't this what Hugging Face wants to do?
Re: How to scale LLMs better with an alternative to transformers
#5I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
This is called ensemble learning
Re: How to scale LLMs better with an alternative to transformers
#6I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.
Re: How to scale LLMs better with an alternative to transformers
#7I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.
This isn't true, GPT4 is not a mixture of experts model.
I'm on a quixotic mission to explain how it became "common knowledge" GPT4 is a trillion parameter mixture of experts model, despite clear denial from OpenAI's CEO. Full recounting: https://news.ycombinator.com/item?id=36828878
Re: How to scale LLMs better with an alternative to transformers
#8I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.
I wouldn't expect C to just select one of the answers A and B have given. But rather to take in information from both answers and come up with a third one which is more than the sum of its parts.
Re: How to scale LLMs better with an alternative to transformers
#9Earlier quoted context omitted.
ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.
This isn't true, GPT4 is not a mixture of experts model. I'm on a quixotic mission to explain how it became "common knowledge" GPT4 is a trillion parameter mixture of experts model, despite clear denial from OpenAI's CEO. Full recounting: https://news.ycombinator.com/item?id=36828878
Sam Altman has never denied that GPT-4 is a mixture of experts model. He denied an early rumor that it was a 100 trillion parameter model.[1] The mixture of experts rumor states that GPT-4 is eight 220B models. That's far more plausible than a single 100 trillion model, and the sources (geohotz and Soumith Chintala[2]) have some credibility. But yeah, it's still only a rumor.
[1] https://www.theverge.com/23560328/openai-gpt-4-rumor-release...
[2] https://twitter.com/soumithchintala/status/16712671501017210...
Re: How to scale LLMs better with an alternative to transformers
#10I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.
That interesting.
Could have a federated LLM approach with different orgs owning different LLM specialties.
Commercial arrangement could look like telco’s roaming agreements.