Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to scale LLMs better with an alternative to transformers

hazyresearch.stanford.edu

Post reply on HN
Print view View original

11–20 of 35 posts

Re: How to scale LLMs better with an alternative to transformers

#11

Post by api »

Dwolb wrote: post #10
viraptor wrote: post #6

Earlier quoted context omitted.

ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.

That interesting. Could have a federated LLM approach with different orgs owning different LLM specialties. Commercial arrangement could look like telco’s roaming agreements.

Could also work in DIY-land with P2P networks of people with different models running.

Re: How to scale LLMs better with an alternative to transformers

#12

Post by [deleted] »

api wrote: post #11
Dwolb wrote: post #10

Earlier quoted context omitted.

That interesting. Could have a federated LLM approach with different orgs owning different LLM specialties. Commercial arrangement could look like telco’s roaming agreements.

Could also work in DIY-land with P2P networks of people with different models running.

[deleted]

Re: How to scale LLMs better with an alternative to transformers

#13

Post by refulgentis »

Sunhold wrote: post #9
refulgentis wrote: post #7

Earlier quoted context omitted.

This isn't true, GPT4 is not a mixture of experts model. I'm on a quixotic mission to explain how it became "common knowledge" GPT4 is a trillion parameter mixture of experts model, despite clear denial from OpenAI's CEO. Full recounting: https://news.ycombinator.com/item?id=36828878

Sam Altman has never denied that GPT-4 is a mixture of experts model. He denied an early rumor that it was a 100 trillion parameter model.[1] The mixture of experts rumor states that GPT-4 is eight 220B models. That's far more plausible than a single 100 trillion model, and the sources (geohotz and Soumith Chintala[2]) have some credibility. But yeah, it's still only a rumor. [1] https://www.theverge.com/23560328/ope…

Read this as if I'm smiling and shaking my head. I'm not upset, I call it a quixotic quest because there's little chance of correcting it given how far it diffused, how few people understand the nuts and bolts, and by far the biggest factor IMHO: confirmation bias.

You cited geohot as an expert on OpenAI[1], and to indicate skepticism Altman denied it, you fixated on the # of parameters, cited a Verge link to a chart in a random tweet about 100 trillion parameters, that it didn't show Sam Altman, and it didn't ask Altman about 100 trillion parameters specifically. And if it did, what does that have to do with mixture of experts?

I flipped to 3 to -2 within 30 minutes of you posting this.

"A lie gets halfway around the world before the truth has a chance to get its pants on." - Churchill

[1] never worked at OpenAI, no notable domain expertise, and a Twitter intern in 2022.

Re: How to scale LLMs better with an alternative to transformers

#14

Post by chaxor »

mg wrote: post #2

I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…

This will perform worse in many cases, better in some cases. There is a lot of knowledge that can be transferred between datasets.

For example, "describe to me if this Amazon product is likely to have stronger tensile strength and if its materials are more safe?" requires knowledge not only from a database of Amazon products and their descriptions, but in this case leaving out knowledge from physics textbooks could be detrimental. Ultimately, these are the types of problems we want these systems to excel at as well, so it's important to access all of the training data. MoE is still a decent idea (can help transfer some of the knowledge between models with a model on top of others), but in order to not get wildly conflicting and/or unrelated stories from each model, some overlap is needed to provide a clearer story to the top model.

Re: How to scale LLMs better with an alternative to transformers

#15

Post by toxik »

viraptor wrote: post #6
mg wrote: post #2

I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…

ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.

That’s not similar at all, actually.

Re: How to scale LLMs better with an alternative to transformers

#16

Post by mg »

chaxor wrote: post #14
mg wrote: post #2

I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…

This will perform worse in many cases, better in some cases. There is a lot of knowledge that can be transferred between datasets. For example, "describe to me if this Amazon product is likely to have stronger tensile strength and if its materials are more safe?" requires knowledge not only from a database of Amazon products and their descriptions, but in this case leaving out knowledge from physics textbooks could b…

Depends.

If A answers "This toaster is made of plastique and paper, one would have to look up their tensile strength to answer your question"

And B answers "I don't know what materials this toaster is made of, but the best tensile strength in toasters is reached when using iron, ok tensil strength is achieved by using copper. One should avoid plastique and paper as these have very bad tensil strenght"

Then C could imply that the tensil strength of that toaster is not good.

Re: How to scale LLMs better with an alternative to transformers

#17

Post by PeterisP »

mg wrote: post #2

I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…

The idea of decentralized hierarchical LLMs is interesting but your chosen example is not a good illustration as all three of these data sources are small and insufficient, any model trained solely on any of them will not be a good model for anything. Other things being equal, data quality and domain matters a lot, but a hundredfold increase in data quantity makes an even larger difference.

Datasets like those can be used for fine tuning a pretrained LLM towards a specific domain, but for decent (not even state of art, just anything usable) results you need a large enough dataset to learn English and general world knowledge, and for that the preferable size is "almost everything you can get your hands on", as in, the quantity you'd want to train on is larger than the quantity of good data you can realistically get. Like, the 800 GiB of text at https://pile.eleuther.ai/ is a good start, but if you could get ten times more data (as some of the big companies probably do, since they have access to lots of user-generated non-public text), you should definitely use that.

If you want targeted LLMs then IMHO the proper mindset for data choice is "take everything that you can out of what humanity has ever written and then pick out of that the most suitable 20% for your needs" and that would give much better results than any single dataset that's only Wikipedia-sized.

Re: How to scale LLMs better with an alternative to transformers

#18

Post by letmevoteplease »

refulgentis wrote: post #13
Sunhold wrote: post #9

Earlier quoted context omitted.

Sam Altman has never denied that GPT-4 is a mixture of experts model. He denied an early rumor that it was a 100 trillion parameter model.[1] The mixture of experts rumor states that GPT-4 is eight 220B models. That's far more plausible than a single 100 trillion model, and the sources (geohotz and Soumith Chintala[2]) have some credibility. But yeah, it's still only a rumor. [1] https://www.theverge.com/23560328/ope…

Read this as if I'm smiling and shaking my head. I'm not upset, I call it a quixotic quest because there's little chance of correcting it given how far it diffused, how few people understand the nuts and bolts, and by far the biggest factor IMHO: confirmation bias. You cited geohot as an expert on OpenAI[1], and to indicate skepticism Altman denied it, you fixated on the # of parameters, cited a Verge link to a chart…

Here is the timeline again:

2022/11/11: A viral tweet claims GPT-4 will have "100 trillion parameters."[1] At this point, there were no rumors about mixture of experts.

2023/01/16: In an interview, Sam Altman mentions he saw the tweet and it was "complete bullshit."[2]

2023/06/20: geohotz and the lead of PyTorch, two people who would be expected to have relevant connections, claim that GPT-4 is an 8 x 220B mixture of experts model.[3]

These are two separate, unconnected rumors. One was denied by Sam Altman and was never plausible in the first place. The other was never denied and is highly plausible. You are conflating them by claiming, without any source, that there was "a clear denial from OpenAI's CEO" that "GPT4 is a trillion parameter mixture of experts model."

[1] https://twitter.com/andrewsteinwold/status/15948895625260277...

[2] https://youtu.be/ebjkD1Om4uw?t=313

[3] https://twitter.com/soumithchintala/status/16712671501017210...

Re: How to scale LLMs better with an alternative to transformers

#19

Post by freilanzer »

chaxor wrote: post #14
mg wrote: post #2

I wonder how a decentralized, hierarchical LLM would perform. For example: LLM A is trained on all of Wikipedia LLM B is trained on all of Hacker News LLM C is trained on all of Project Gutenberg User asks question Q on webservice W. W sends Q to A and B. Then W sends a question to C "Hey C, I have a user who asked Q. Here is A's reply and B's reply. Given those, how would you answer Q?" Would the answer be as good a…

This will perform worse in many cases, better in some cases. There is a lot of knowledge that can be transferred between datasets. For example, "describe to me if this Amazon product is likely to have stronger tensile strength and if its materials are more safe?" requires knowledge not only from a database of Amazon products and their descriptions, but in this case leaving out knowledge from physics textbooks could b…

This might suggest that it works: https://viterbischool.usc.edu/news/2023/07/teaching-robots-t...

Re: How to scale LLMs better with an alternative to transformers

#20

Post by _ea1k »

refulgentis wrote: post #7
viraptor wrote: post #6

Earlier quoted context omitted.

ChatGPT-4 does something a bit similar with the mixture-of-experts approach. Although if I understand it correctly, they select which networke to use ahead of time rather than select the best answer from multiple.

This isn't true, GPT4 is not a mixture of experts model. I'm on a quixotic mission to explain how it became "common knowledge" GPT4 is a trillion parameter mixture of experts model, despite clear denial from OpenAI's CEO. Full recounting: https://news.ycombinator.com/item?id=36828878

> This isn't true, GPT4 is not a mixture of experts model.

I don't know if you are right or not, but I've been shocked at how quickly people flipped to just accepting that GPT4 was a mixture of experts model given the scant evidence to support the claim.

It is possible, but not particularly likely.

Post reply on HN