Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Not So Mysterious or Different

arxiv.org

Post reply on HN
Print view View original

121–130 of 139 posts

Re: Deep Learning Is Not So Mysterious or Different

#121

Post by theahura »

woopwoop wrote: post #119

A decade ago the paper "Understanding deep learning requires rethinking generalization" [0] was published. The submission is a response to that paper and subsequent literature. Deep neural nets are notable for their strong generalization performance: despite being highly overparametrized they do not seem to overfit the training data. They still perform well on hold-out data and very often on out of distribution data…

Doesn't the simplicity bias come explicitly from regularization techniques, including drop out or l2 norm?

Re: Deep Learning Is Not So Mysterious or Different

#122

Post by woopwoop »

theahura wrote: post #121
woopwoop wrote: post #119

A decade ago the paper "Understanding deep learning requires rethinking generalization" [0] was published. The submission is a response to that paper and subsequent literature. Deep neural nets are notable for their strong generalization performance: despite being highly overparametrized they do not seem to overfit the training data. They still perform well on hold-out data and very often on out of distribution data…

Doesn't the simplicity bias come explicitly from regularization techniques, including drop out or l2 norm?

Those are not necessary to reproduce benign overfitting

Re: Deep Learning Is Not So Mysterious or Different

#123

Post by uoaei »

bonoboTP wrote: post #105
uoaei wrote: post #59

Earlier quoted context omitted.

Sure. "Preprint" implies prior to printing, which means that there's a reasonable expectation for this paper to be submitted, accepted, and printed in a scholarly journal. What we have here is little more than a tongue-in-cheek submission which carries an aesthetic of "hot-take" throughout the paper. This is unbecoming of one committed to scholarly pursuits and all but guarantees rejection from journals committed to…

This is still just name calling. You are just using negatively charged adjectives without quoting or arguing the substance or even the style. Is your crique only about the presentation or the substance of the ideas too? What makes it unprofessional? To me it looks much better than a substantial chunk of my review stacks at ML conferences and journals. Are you an ML researcher? Maybe you're used to a different researc…

You're making a normative argument. The fact that other people publish crap is irrelevant, unless you actually intend to lend implicit justification of the status quo's existence just because it exists. "Ought", meet "is", etc.

The take proferred by TFA just isn't a useful take at all except perhaps for those who have never been elbow-deep in ML model architecture design, analysis, and training. The headline alludes to a surprising fact that you learn throughout course studies, a sidenote that can be repeatedly referred back to in order to emphasize the universality of statistical reasoning, but it's certainly not worthy of some kind of manifesto.

I agree we need to demystify ML for the common audience but this is a messaging problem much moreso than it is a pedagogical one. Typically the standard for publication is "genuine novel contribution" but no one who has been through a study regimen about ML will learn anything new from this. Preprints are supposed to be reserved for those papers which anticipate publication but I see no path for this paper to be accepted anywhere.

Re: Deep Learning Is Not So Mysterious or Different

#124

Post by SJC_Hacker »

procaryote wrote: post #99
nickysielicki wrote: post #26

Earlier quoted context omitted.

Markov chains are very very far off from gpt2.

Aren't they technically the same? GPT picks the next token given the state of current context, based on probabilities and a random factor. That is mathematically equivalent to a Markov chain, isn't it?

Markov chains don't account for the full history. While all LLMs do have a context length, this is more a practical limitation based on resources rather than anything implicit in the model.

Re: Deep Learning Is Not So Mysterious or Different

#125

Post by bonoboTP »

uoaei wrote: post #123
bonoboTP wrote: post #105

Earlier quoted context omitted.

This is still just name calling. You are just using negatively charged adjectives without quoting or arguing the substance or even the style. Is your crique only about the presentation or the substance of the ideas too? What makes it unprofessional? To me it looks much better than a substantial chunk of my review stacks at ML conferences and journals. Are you an ML researcher? Maybe you're used to a different researc…

You're making a normative argument. The fact that other people publish crap is irrelevant, unless you actually intend to lend implicit justification of the status quo's existence just because it exists. "Ought", meet "is", etc. The take proferred by TFA just isn't a useful take at all except perhaps for those who have never been elbow-deep in ML model architecture design, analysis, and training. The headline alludes…

The paper offers a counterpoint to a published work (Zhang et al., 2021) which together with their earlier unpublished Arxiv version from 2016 has over 7000 citations. If you disagree with this rebuttal, by all means formulate what you find lacking.

> The fact that other people publish crap is irrelevant

You argued that this work is not something that can be seriously be considered to be submitted for publication and cannot be counted as a preprint. It has been pointed out that many works do get submitted to academic venues that aren't up to this quality. You're shifting goalposts.

You are making dismissive remarks without having to state your own view. Do you think Zhang et al's view is correct and deep learning shows novel effects that existing tools can't describe? Do you think the current manuscript does not effectively address those points? You have to argue if you think you have arguments. Labeling something a manifesto or a hot take is just low effort jab. Why do you think that the paper has no chance of acceptance? You are rehashing the same non-argument in different words.

A useful comment would state something like: the authors still do not explain effect X and Y that appears only in deep learning and not in classic ML. Or: the authors' point regarding effect X is incorrect and does not actually show what they claim to show. Etc. Simply saying "it's unserious" can be just turned back at your comment the same way.

Re: Deep Learning Is Not So Mysterious or Different

#126

Post by tzs »

amelius wrote: post #108
randomtoast wrote: post #35

Earlier quoted context omitted.

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

Nonsense. Chatgpt uses the word a lot precisely because people used it a lot.

Apparently this depends on where people are. It is not used a lot in US English, but it is used a lot in African English.

Part of training LLMs involves extensive human feedback, and many LLM makers outsource that to Africa to save money. The LLMs then pick up and use African English.

See the link in this comment [1] for an interesting article about this.

[1] https://news.ycombinator.com/item?id=43394220

Re: Deep Learning Is Not So Mysterious or Different

#127

Post by [deleted] »

godelski wrote: post #62
tasty_freeze wrote: post #56

Earlier quoted context omitted.

That seems utterly bizarre to me. I don't use "delve" frequently myself, but it is common enough that it doesn't jump out as an unusual word. Perhaps it is overused or used in a not-exactly-usual context that tips one off that it is LLM-generated, but by itself it signifies nothing to me.

It is a very common word used in Nigerian style English which was a very common place they were outsourcing RLHF tasks to. A sibling comment has a link but it is also easy to google.

[deleted]

Re: Deep Learning Is Not So Mysterious or Different

#128

Post by bogeholm »

rottc0dd wrote: post #85
bogeholm wrote: post #54

Earlier quoted context omitted.

Looks nice - are there written versions?

There is a course reader for CS109 [1]. You can download pdf version of this. There is also book[2] for excellent caltech course[3]. [1] https://chrispiech.github.io/probabilityForComputerScientist... [2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/... [3] https://work.caltech.edu/telecourse

Thanks!

Re: Deep Learning Is Not So Mysterious or Different

#129

Post by sharpshadow »

randomtoast wrote: post #35
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

For me it’s “eerie” it just will not stop using this word.

Re: Deep Learning Is Not So Mysterious or Different

#130

Post by uoaei »

bonoboTP wrote: post #125
uoaei wrote: post #123

Earlier quoted context omitted.

You're making a normative argument. The fact that other people publish crap is irrelevant, unless you actually intend to lend implicit justification of the status quo's existence just because it exists. "Ought", meet "is", etc. The take proferred by TFA just isn't a useful take at all except perhaps for those who have never been elbow-deep in ML model architecture design, analysis, and training. The headline alludes…

The paper offers a counterpoint to a published work (Zhang et al., 2021) which together with their earlier unpublished Arxiv version from 2016 has over 7000 citations. If you disagree with this rebuttal, by all means formulate what you find lacking. > The fact that other people publish crap is irrelevant You argued that this work is not something that can be seriously be considered to be submitted for publication and…

> many works do get submitted to academic venues that aren't up to this quality

There's that normative argument rearing its head again. Not interested in jumping off bridges just because your friends do it, thanks.

It is unserious. The thesis amounts to "statistical models like these are mean-field, roughly-max-entropy approximations under the implied data generating process" which is not only an offhand comment a professor might make in ML 201 but tautological on its face. The fact they drag in a couple citations to say as much is besides the point entirely.

Post reply on HN