I am of the general understanding that this paper became less about the LLMs & more of a insinuating hit piece against Alphabet. At least, some of the controversial nuggets got Gebru (and later M Mitchell) fired. From a technical standpoint, there is little new stuff that I found this paper offered in understanding why LLMs can have unpredictable nature, or what degree of data will get exposed by clever hacks (or if…
On the dangers of stochastic parrots: Can language models be too big? (2021)
71–80 of 111 posts
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#72I believe this is the papers that got timnit and mmitchel fired from google, followed by a protracted media/legal campaign against google and vice versa.
A small correction: this paper didn't get her fired, her reaction to feedback on this paper got her fired. Note to all: if you give an employer an ultimatum "do X or I resign", don't be surprised if they accept your resignation.
[deleted]
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#73I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and…
> I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN.
I haven't read the paper beyond the abstract yet, so can't comment on it's contents, but can you specify what you mean by "how people have described it on HN"?
To be frank, I don't like this style of argument, because I literally don't know what you're referring to. I'm quite familiar with the hubbub that happened when Gebru left Google, but feelings around what happened there shouldn't be misconstrued as feedback on this paper specifically.
In other words, if you think that people are arguing that this paper is "outside the realm of valid academic discourse", I think you should call out what you're referring to, because I haven't seen that as a widely held opinion. It feels like a straw man, or at the very least you are discounting some very specific critiques other commenters have made by just broadly referring to "HN sentiment".
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#74Earlier quoted context omitted.
When you hard code in a blacklist is that really considered progress?
Yes. Machine learning models learn from the data they are fed. Thus, they end up with the same biases that humans have. There is no "natural" fix to this, as we are naturally biased. And even worse, we don't even all agree on a single set of moral values. Thus, any techniques aiming to eliminate bias must come in the form of a set of hard coded definitions of what the author feels is the correct set of morals. Curren…
[deleted]
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#75Earlier quoted context omitted.
I suspect it was Timnit’s behavior after the paper didn’t pass internal review that actually got her fired (issuing an ultimatum and threatening to resign unless the company met her demands; telling her coworkers to stop writing documents because their work didn’t matter; insinuations of racist/misogynistic treatment from leadership when she didn’t get her way).
It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" pr…
I’m sorry but this is a bunch of crock and honestly sounds like just a lot more speculation adding nothing to the conversation.
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#76Earlier quoted context omitted.
I suspect it was Timnit’s behavior after the paper didn’t pass internal review that actually got her fired (issuing an ultimatum and threatening to resign unless the company met her demands; telling her coworkers to stop writing documents because their work didn’t matter; insinuations of racist/misogynistic treatment from leadership when she didn’t get her way).
It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" pr…
[deleted]
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#77Earlier quoted context omitted.
It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" pr…
I’m sorry but this is a bunch of crock and honestly sounds like just a lot more speculation adding nothing to the conversation.
Nah, this is the actual truth. You can feel free not to believe me, but I have more complete information about this situation than anyone else you'll ever talk to.
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#78I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and…
I guess if one aspires to academic discourse about Truth rather than about Activism or about Power then this will seem somewhat like motivated reasoning.
If you want academic discourse about truth, become a philosopher. Activism and power are just as real as trees and language models. Discussing them is discussing the world.
Your concern about motivated reasoning is a valid one – but motivated reasoning is different to persuasive language. If anything, persuasive language highlights the places you should be looking for motivated reasoning (but it also makes it harder to spot, for some people; I don't recommend using persuasive language in your own papers).
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#79I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and…
The sin is not activist language, the sin is applying rhetoric in academic language to distort the truth to make a point. It’s concern trolling to gain academic clout. Page 3 of a 14 page paper already claims with a straight face that LLMs are responsible for the sinking of Maldives. > Training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to require as much energy as a trans-American…
[flagged]
Re: On the dangers of stochastic parrots: Can language models be too big? (2021)
#80I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and…
The sin is not activist language, the sin is applying rhetoric in academic language to distort the truth to make a point. It’s concern trolling to gain academic clout. Page 3 of a 14 page paper already claims with a straight face that LLMs are responsible for the sinking of Maldives. > Training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to require as much energy as a trans-American…
The text you quoted claims that the "environmental price" is of "as much energy as a trans-American flight". It implies that this would contribute to these issues (true), and rhetorically suggests that this is not fair, but words-as-written, I don't see any distortion of truth. Heck, it's using familiar units: most of us have an order-of-magnitude idea how many trans-American flights there are each month.
"Author is not affecting a neutral point of view" is not the same as "author is distorting truth".