...from the same team that brought you FlashAttention, S4, H3, and Hyena.
As always, we have to wait until this has been tested at much larger scale.
How to scale LLMs better with an alternative to transformers
21–30 of 35 posts
Re: How to scale LLMs better with an alternative to transformers
#22Earlier quoted context omitted.
Sam Altman has never denied that GPT-4 is a mixture of experts model. He denied an early rumor that it was a 100 trillion parameter model.[1] The mixture of experts rumor states that GPT-4 is eight 220B models. That's far more plausible than a single 100 trillion model, and the sources (geohotz and Soumith Chintala[2]) have some credibility. But yeah, it's still only a rumor. [1] https://www.theverge.com/23560328/ope…
Read this as if I'm smiling and shaking my head. I'm not upset, I call it a quixotic quest because there's little chance of correcting it given how far it diffused, how few people understand the nuts and bolts, and by far the biggest factor IMHO: confirmation bias. You cited geohot as an expert on OpenAI[1], and to indicate skepticism Altman denied it, you fixated on the # of parameters, cited a Verge link to a chart…
[flagged]
Re: How to scale LLMs better with an alternative to transformers
#23I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
Isn't that more or less how GPT-4 works? multiple "expert" LLMs giving input depending on the context?[0]
[0]https://the-decoder.com/gpt-4-architecture-datasets-costs-an...
the biggest issue is if you have too many specialists and spin a lot of them to reply to the same query and after that discard the less optimal answers.
Your answer quality might improve, but the computing costs could skyrocket without some smart filtering and distribution before you reach any LLM
Re: How to scale LLMs better with an alternative to transformers
#24Could this be used in conjunction with sbert to get better performing sentence_transformers for longer sequences?
Re: How to scale LLMs better with an alternative to transformers
#25Earlier quoted context omitted.
Read this as if I'm smiling and shaking my head. I'm not upset, I call it a quixotic quest because there's little chance of correcting it given how far it diffused, how few people understand the nuts and bolts, and by far the biggest factor IMHO: confirmation bias. You cited geohot as an expert on OpenAI[1], and to indicate skepticism Altman denied it, you fixated on the # of parameters, cited a Verge link to a chart…
Here is the timeline again: 2022/11/11: A viral tweet claims GPT-4 will have "100 trillion parameters."[1] At this point, there were no rumors about mixture of experts. 2023/01/16: In an interview, Sam Altman mentions he saw the tweet and it was "complete bullshit."[2] 2023/06/20: geohotz and the lead of PyTorch, two people who would be expected to have relevant connections, claim that GPT-4 is an 8 x 220B mixture of…
1. You did find a tweet that claimed 100 trillion parameters, as the GP post did.
2. The video mentions he saw _a_ tweet about GPT...and actually we don't even know what the tweet said, the moderator never finished their question.
3. I'm not sure what sort of claim "connected" is, other than unfalsifiable, like all of the confirmation bias motivated arguing on this topic. People do know Geohot's name and Pytorch is an open source ML framework, neither of which make them likely venues to know a closely kept trade secret of Open AI's. (and as we show in the rest of this post, they were parroting claims made months earlier, I'm showing you through March '23, Geohot didn't get around to repeating it until June!)
Recentering: it's not a mixture of experts model, no matter if people claimed 1 trillion, 100 trillion or both. (btw, easy proof of the extensive 1 trillion claims: innumerable, all in 2022: https://twitter.com/search?q=until%3A2022-12-31%20since%3A20...)
Now: let's say a reader just can't let go of the fact some people also made 100 trillion claims, but I said most people made 1 trillion claims. I'm not sure what to say, because I never claimed no one made 100 trillion claims as well, so I'm not sure how to give those people peace so we can talk mixture of experts. I guess apologize? I'm sorry.
Now we can definitely focus on mixture of experts.
Here's innumerable claims between Jan 1st 2023 and March 31st 2023 that GPT4 was a 1 trillion mixture of experts model, as I claimed: [https://www.google.com/search?q=mixture+of+experts+trillion+.... [/r/MachineLearning](https://www.reddit.com/r/MachineLearning/comments/121q6nk/n_...) [the-decoder](https://the-decoder.com/gpt-4-has-a-trillion-parameters/) [rando boards](https://www.futuretimeline.net/forum/viewtopic.php?p=31145)
Re: How to scale LLMs better with an alternative to transformers
#26Interesting! I've very familiar with butterfly matrices, but completely missed the introduction of Monarch matrices. I'm excited to unpack these definitions later.
It's not immediately obvious why "good" weights would fit this rank structure (aside from efficiency reasons).
Re: How to scale LLMs better with an alternative to transformers
#27I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
The idea of decentralized hierarchical LLMs is interesting but your chosen example is not a good illustration as all three of these data sources are small and insufficient, any model trained solely on any of them will not be a good model for anything. Other things being equal, data quality and domain matters a lot, but a hundredfold increase in data quantity makes an even larger difference. Datasets like those can be…
Have you seen the recent work at TinyStories:
- https://arxiv.org/abs/2305.07759
It got some nice attention here: - https://github.com/karpathy/llama2.c
I think there may be some applications in this limited space that are worth looking into. You won’t replicate GPT-anything but it may be possible to solve some nice problems very much more efficiently that one would expect at first.
Re: How to scale LLMs better with an alternative to transformers
#28Had to laugh at this sample output:
> "dataset":"oasst",
> "instruction":"What do you think about ChatGPT?",
> "output":"ChatGPT is a chatbot developed by Meta AI…
Re: How to scale LLMs better with an alternative to transformers
#29Interesting! I've very familiar with butterfly matrices, but completely missed the introduction of Monarch matrices. I'm excited to unpack these definitions later. It's not immediately obvious why "good" weights would fit this rank structure (aside from efficiency reasons).
This is moving so fast
Re: How to scale LLMs better with an alternative to transformers
#30I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…
The idea of decentralized hierarchical LLMs is interesting but your chosen example is not a good illustration as all three of these data sources are small and insufficient, any model trained solely on any of them will not be a good model for anything. Other things being equal, data quality and domain matters a lot, but a hundredfold increase in data quantity makes an even larger difference. Datasets like those can be…
That is not so certain. Microsoft's "Textbooks are all you need" is a case in point. https://news.ycombinator.com/item?id=36413768