Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding large language models: A cross-section of the relevant literature

magazine.sebastianraschka.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Understanding large language models: A cross-section of the relevant literature

#11

Post by SanderNL »

notnaut wrote: post #10
atleastoptimal wrote: post #9

I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years. The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.

So you think the logical patterns found in human language might also be similar enough to the logical patterns found in other systems that these LLMs have a jumpstart in figuring those out? We already have such a poor idea of how these things seem to understand so much… it’ll be a wild day when cancer is cured and we have absolutely no idea why.

It feels like some type of Heisenbergian deal. You can solve what you want, but you cannot know how at the same time.

Re: Understanding large language models: A cross-section of the relevant literature

#12

Post by atleastoptimal »

notnaut wrote: post #10
atleastoptimal wrote: post #9

I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years. The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.

So you think the logical patterns found in human language might also be similar enough to the logical patterns found in other systems that these LLMs have a jumpstart in figuring those out? We already have such a poor idea of how these things seem to understand so much… it’ll be a wild day when cancer is cured and we have absolutely no idea why.

IMO a sufficiently advanced AI should be able to do a full analysis of its neural architecture and explain it and break down its functionality.

Re: Understanding large language models: A cross-section of the relevant literature

#14

Post by astrange »

atleastoptimal wrote: post #12
notnaut wrote: post #10

Earlier quoted context omitted.

So you think the logical patterns found in human language might also be similar enough to the logical patterns found in other systems that these LLMs have a jumpstart in figuring those out? We already have such a poor idea of how these things seem to understand so much… it’ll be a wild day when cancer is cured and we have absolutely no idea why.

IMO a sufficiently advanced AI should be able to do a full analysis of its neural architecture and explain it and break down its functionality.

There is no capability in an LLM to do this and we don't know how to build an "AI" that is not an LLM.

(Just deciding that your LLM has magic powers because you've put it in a category called "AI" and you've decided that category has said magic powers is what that guy Wittgenstein was complaining about in philosophical problems. Besides, intelligence doesn't mean all your thoughts are automatically correct!)

Re: Understanding large language models: A cross-section of the relevant literature

#15

Post by astrange »

notnaut wrote: post #10
atleastoptimal wrote: post #9

I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years. The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.

So you think the logical patterns found in human language might also be similar enough to the logical patterns found in other systems that these LLMs have a jumpstart in figuring those out? We already have such a poor idea of how these things seem to understand so much… it’ll be a wild day when cancer is cured and we have absolutely no idea why.

You can't "cure cancer" because "cancer" is a label for different things with different causes. If your brain cells all have the wrong DNA there's no way to get different brain cells.

Re: Understanding large language models: A cross-section of the relevant literature

#16

Post by algo_trader »

jdaggers wrote: post #13

I've been doing research with NLP for about 5 years now I couldn't have created a better inrto list than this. Many of these resources were things I've sent people over the years.

it IS a great list

Do BERTs models learn faster than GPTs?

Assuming the task is predictive, BERT seem to have a stronger/richer signal (obviously in practice you have use existing pre-trained models)

Re: Understanding large language models: A cross-section of the relevant literature

#17

Post by azinman2 »

simonw wrote: post #5
two_in_one wrote: post #4

Earlier quoted context omitted.

The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.

It will never completely replace Google search because no matter how large the model gets it will always have a training cut-off date in the past, and there will always be specific factual information that isn't a good fit for language models. If you're looking for a LLM replacement for regular Google search then Bing and Bard are much more likely to fit the bill.

It won’t replace google search because ultimately it doesn’t have a point of view, let alone a dedicated page with a variety of functions (although plugins eat somewhat at this). For example, pick a random article written for The Atlantic — we are getting human prose from a specific author with a specific writing style about a specific subject that fits the zeitgeist of the day with a specific length suitable for what’s being communicated. That is something that is both enjoyable and informative, with style. LLMs are great at summarizing and synthesizing, but it’s not the same thing.

But perhaps the LLM can eventually point me towards this to then learn more; somewhat what Bing is already doing but it’s more of a footnote than something upfront (e.g. “you might want to read an excellent write up in The Atlantic about decriminalizing drugs”).

Re: Understanding large language models: A cross-section of the relevant literature

#18

Post by bambataa »

astrange wrote: post #14
atleastoptimal wrote: post #12

Earlier quoted context omitted.

IMO a sufficiently advanced AI should be able to do a full analysis of its neural architecture and explain it and break down its functionality.

There is no capability in an LLM to do this and we don't know how to build an "AI" that is not an LLM. (Just deciding that your LLM has magic powers because you've put it in a category called "AI" and you've decided that category has said magic powers is what that guy Wittgenstein was complaining about in philosophical problems. Besides, intelligence doesn't mean all your thoughts are automatically correct!)

> Besides, intelligence doesn't mean all your thoughts are automatically correct!)

This is the true bitter lesson for HN.

Re: Understanding large language models: A cross-section of the relevant literature

#19

Post by rimliu »

atleastoptimal wrote: post #9

I think LLMs are gonna be the "jumpstart" of more general AGI prototypes over the next couple years. The large corpus of text gives them a general basis of logical patterns, which can then be pruned iteratively in simulated environments.

Give computers the alphabet. It contains all the texts. Imagine the possibilities.

Re: Understanding large language models: A cross-section of the relevant literature

#20

Post by sampo »

two_in_one wrote: post #4
cs702 wrote: post #2

This is a good intro for anyone who already has at least some background in ML and wants to get up to speed on LLMs relatively quickly. Props to the author for giving credit to Bandanau et al (2014), which I believe first proposed the concept of applying a Softmax function over token scores to compute attention, setting the stage for the original transformer by Vaswani et al (2017).

The big question here is what's next? So far my coding experiments with GPT-4 shows it's a shallow thinker. It has ready answers for many questions. One step further and it fails miserably. Much better but not a full replacement for google search.

> The big question here is what's next?

These lecture slides from Yann LeCun from 1 month ago. Starting from slide 13, he presents his idea of a roadmap towards autonomous machine intelligence.

https://drive.google.com/file/d/1BU5bV3X5w65DwSMapKcsr0ZvrMR...

Post reply on HN