Understanding large language models: A cross-section of the relevant literature
magazine.sebastianraschka.com
Understanding large language models: A cross-section of the relevant literature
1–10 of 32 posts
Re: Understanding large language models: A cross-section of the relevant literature
#2This is a good intro for anyone who already has at least some background in ML and wants to get up to speed on LLMs relatively quickly.
Props to the author for giving credit to Bandanau et al (2014), which I believe first proposed the concept of applying a Softmax function over token scores to compute attention, setting the stage for the original transformer by Vaswani et al (2017).
Re: Understanding large language models: A cross-section of the relevant literature
#3Two other recent literature reviews worth reading:
"Transformer Taxonomy" - https://kipp.ly/blog/transformer-taxonomy/
"Five years of progress in GPTs" - https://finbarrtimbers.substack.com/p/five-years-of-progress...
Re: Understanding large language models: A cross-section of the relevant literature
#4This is a good intro for anyone who already has at least some background in ML and wants to get up to speed on LLMs relatively quickly. Props to the author for giving credit to Bandanau et al (2014), which I believe first proposed the concept of applying a Softmax function over token scores to compute attention, setting the stage for the original transformer by Vaswani et al (2017).
The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.
Re: Understanding large language models: A cross-section of the relevant literature
#5This is a good intro for anyone who already has at least some background in ML and wants to get up to speed on LLMs relatively quickly. Props to the author for giving credit to Bandanau et al (2014), which I believe first proposed the concept of applying a Softmax function over token scores to compute attention, setting the stage for the original transformer by Vaswani et al (2017).
The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.
It will never completely replace Google search because no matter how large the model gets it will always have a training cut-off date in the past, and there will always be specific factual information that isn't a good fit for language models.
If you're looking for a LLM replacement for regular Google search then Bing and Bard are much more likely to fit the bill.
Re: Understanding large language models: A cross-section of the relevant literature
#6This is a good intro for anyone who already has at least some background in ML and wants to get up to speed on LLMs relatively quickly. Props to the author for giving credit to Bandanau et al (2014), which I believe first proposed the concept of applying a Softmax function over token scores to compute attention, setting the stage for the original transformer by Vaswani et al (2017).
The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.
Try https://www.phind.com/. I find it's much better optimized for coding purposes than regular GPT-4.
Re: Understanding large language models: A cross-section of the relevant literature
#7Earlier quoted context omitted.
The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.
It will never completely replace Google search because no matter how large the model gets it will always have a training cut-off date in the past, and there will always be specific factual information that isn't a good fit for language models. If you're looking for a LLM replacement for regular Google search then Bing and Bard are much more likely to fit the bill.
I think incremental training should be possible. When next model is much smaller and as an input takes latent state from the bigger model. Then training will be faster, those smaller models can be chained.
Another option is to have small re-trainable areas within the big model.
Of course ideally model should be able to work with some sort of "knowledge base", which can be updated daily. As a main, or additional data source.
As for me I'm looking for ways to increase productivity. Have ChatGPT Plus, and in waiting lists for GPT-4 and Bing. Neither solves big problems, but usually there are many small things which can be done faster. I would rather offload them.
Re: Understanding large language models: A cross-section of the relevant literature
#8Earlier quoted context omitted.
The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.
Try https://www.phind.com/ . I find it's much better optimized for coding purposes than regular GPT-4.
Just tried it. I like that it lists the information sources in the right-hand panel, and provides the digested information in the main panel
Re: Understanding large language models: A cross-section of the relevant literature
#9I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years.
The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.
Re: Understanding large language models: A cross-section of the relevant literature
#10I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years. The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.
So you think the logical patterns found in human language might also be similar enough to the logical patterns found in other systems that these LLMs have a jumpstart in figuring those out?
We already have such a poor idea of how these things seem to understand so much… it’ll be a wild day when cancer is cured and we have absolutely no idea why.