Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's Going on in Machine Learning? Some Minimal Models

writings.stephenwolfram.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: What's Going on in Machine Learning? Some Minimal Models

#61

Post by lossolo »

GaggiX wrote: post #58
wredue wrote: post #57

Earlier quoted context omitted.

It absolutely is NOT what humans are doing. When humans write, they are serializing thoughts. Humans (well, most of us. Certainly not AI enthusiasts), are reasoning and thinking. When AI writes, it is following a mathematical pathway to string words together that it has seen together before in the given context.

When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model. And if it wasn't obvious, an LLM can string together two words that it had never seen together in the training dataset, it really shows how people ten…

> When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model.

Depends on what your definition of a novel problem is. If it's some variation of a problem that has already been seen in some form in the training data, then yes. But if you mean a truly novel problem—one that humans haven't solved in any form before (like the Millennium Problems, a cancer cure, new physics theories, etc.)—then no, LLMs haven't solved a single problem.

> And if it wasn't obvious, an LLM can string together two words that it had never seen together in the training dataset, it really shows how people tend to simplify the extremely complex dynamics by which these models operate.

Well, for anyone who knows how latent space and attention work in transformer models, it's pretty obvious that they can be used together. But I guess for someone who doesn't know the internals, this could seem like magic or reasoning.

Re: What's Going on in Machine Learning? Some Minimal Models

#62

Post by sdenton4 »

wrsh07 wrote: post #37
sdenton4 wrote: post #34

Earlier quoted context omitted.

So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible. A good take-away from the Wo…

But to your point - that is how I feel about graph nns vs transformers or the fully connected set (GPUs are so good at transformers and fully connected nns, even if there is a structure that makes sense we don't have the hardware to have it make sense.... Unless grok makes it cheap??)

Perhaps; in a lot of cases the architecture barely matters. Transformers took a lot of extra tricks to get working well; the ConvNext paper showed that applying those same tricks to convolutional networks can fully close the gap.

https://arxiv.org/abs/2201.03545

Re: What's Going on in Machine Learning? Some Minimal Models

#63

Post by GaggiX »

lossolo wrote: post #61
GaggiX wrote: post #58

Earlier quoted context omitted.

When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model. And if it wasn't obvious, an LLM can string together two words that it had never seen together in the training dataset, it really shows how people ten…

> When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model. Depends on what your definition of a novel problem is. If it's some variation of a problem that has already been seen in some form in the training d…

>then no, LLMs haven't solved a single problem.

Using your definition of a novel problem, do most people solve novel problems? If so, give me an example of a novel problem you have solved.

Re: What's Going on in Machine Learning? Some Minimal Models

#64

Post by taneq »

usgroup wrote: post #9

Tsetlin machines have been around for some time: https://en.wikipedia.org/wiki/Tsetlin_machine They are discrete, individually interpretable, and can be configured into complicated architectures.

For a wiki article, this seems almost deliberately obtuse. What actually is one, in plain language?

Re: What's Going on in Machine Learning? Some Minimal Models

#65

Post by lossolo »

GaggiX wrote: post #63
lossolo wrote: post #61

Earlier quoted context omitted.

> When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model. Depends on what your definition of a novel problem is. If it's some variation of a problem that has already been seen in some form in the training d…

>then no, LLMs haven't solved a single problem. Using your definition of a novel problem, do most people solve novel problems? If so, give me an example of a novel problem you have solved.

Sure, I did—a lot of them. These are the ones that were not in my training dataset in any form before I solved them, as it's impossible for a human to hold all scientific papers, historical facts, and in general, the entirety of human knowledge and experiences from the entire internet in their brain.

Re: What's Going on in Machine Learning? Some Minimal Models

#66

Post by vessenes »

mjburgess wrote: post #30
vessenes wrote: post #13

Classic Wolfram — brilliant, reimplements / comes at a current topic using only cellular automata, and draws some fairly deep philosophical conclusions that are pretty intriguing. The part I find most interesting is his proposal that neural networks largely work by “hitching a ride” on fundamental computational complexity, in practice sort of searching around the space of functions representable by an architecture fo…

> And, to the extent this is true, that puts explainability at fundamental odds with the highest value / most dense / best deep learning outputs — if they are easily “explainable” by inspection, then they are likely not using all of the complexity available to them. Could you define explainability in this context?

The ability of humans, by inspection, to determine why a program is constructed in the way that it is vis-a-vis the goal/output.

Re: What's Going on in Machine Learning? Some Minimal Models

#67

Post by e12e »

nyrikki wrote: post #56
e12e wrote: post #32

Earlier quoted context omitted.

This makes very little sense (as a contrast to chatgpt predicted that the likely continuation of factorial and 5 is 120). Perhaps if you are able to share the chat session it's possible to see if you likely confused the issue with various factorial implementations - or got chatgpt to run your code with 5 as input? I mean the code is redundant: https://chatgpt.com/share/be249097-5067-4e3d-93c7-3eebedb510...

Do a google search with 'before:2020' on that code, that is recall from pre-training, not 'calculating'

I misread gp's comment, we're in agreement.

Re: What's Going on in Machine Learning? Some Minimal Models

#68

Post by GaggiX »

lossolo wrote: post #65
GaggiX wrote: post #63

Earlier quoted context omitted.

>then no, LLMs haven't solved a single problem. Using your definition of a novel problem, do most people solve novel problems? If so, give me an example of a novel problem you have solved.

Sure, I did—a lot of them. These are the ones that were not in my training dataset in any form before I solved them, as it's impossible for a human to hold all scientific papers, historical facts, and in general, the entirety of human knowledge and experiences from the entire internet in their brain.

you haven't given me a concrete example.

Re: What's Going on in Machine Learning? Some Minimal Models

#69

Post by mjburgess »

vessenes wrote: post #66
mjburgess wrote: post #30

Earlier quoted context omitted.

> And, to the extent this is true, that puts explainability at fundamental odds with the highest value / most dense / best deep learning outputs — if they are easily “explainable” by inspection, then they are likely not using all of the complexity available to them. Could you define explainability in this context?

The ability of humans, by inspection, to determine why a program is constructed in the way that it is vis-a-vis the goal/output.

hmm -- could you speak to the meaning of explanation here?

It seems youre talking about the ability of a researcher to diagnose the structure of the model?

Re: What's Going on in Machine Learning? Some Minimal Models

#70

Post by wredue »

GaggiX wrote: post #58
wredue wrote: post #57

Earlier quoted context omitted.

It absolutely is NOT what humans are doing. When humans write, they are serializing thoughts. Humans (well, most of us. Certainly not AI enthusiasts), are reasoning and thinking. When AI writes, it is following a mathematical pathway to string words together that it has seen together before in the given context.

When an LLM solves a novel problem, it's also reasoning, unless you use some contrived definition of the word "reasoning" that doesn't match how the word is actually used in normal conversation. Also I fully expect the human brain to be encoded in a mathematical model. And if it wasn't obvious, an LLM can string together two words that it had never seen together in the training dataset, it really shows how people ten…

No. It is the AI enthusiasts that use contrived, often entirely random definitions of “reasoning”.

AIs do not “think” in any capacity and are therefore incapable of reasoning. However, if you wish to take “thinking” out of the definition, where we allow an AI to try its hand at “novel (for it)” problems, then AIs fail the test horrifically. I agree, they will probably spit something out and sound confident, but sounding confident is not being correct, and AIs tend to not be correct when something truly new to them is thrown at them. AIs spit out straight incorrect answers (colloquially called “hallucinations” so that AI enthusiasts can downplay the fact that it is factually wrong) for things that an AI is heavily trained on.

If we train an AI on what a number is. But then we slap it with 2+2 =5 long enough, it will eventually start to incorrectly state that 2+2=5. Humans, however, due to their capacity to actually think and reason, can confidently tell you, no matter how much you beat them over the head, that 2+2 is 4 because that’s how numbers work.

Even if we somehow get a human to state that 2+2=5 as an actual thought pattern, they would be capable of reasoning out the problems the moment we start asking “what about 2+3?” Where an AI might make the connection, but there no forward thinking won’t resolve the issue.

Post reply on HN