Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

About Google's approach to research publication – Jeff Dean

docs.google.com

Post reply on HN
Print view View original

111–120 of 607 posts

Re: About Google's approach to research publication – Jeff Dean

#111

Post by aphyr »

Dean writes:

Our aim is to rival peer-reviewed journals in terms of the rigor and thoughtfulness in how we review research before publication.

But Gebru writes that HR and her management chain delivered her feedback in a surprise meeting where she was not allowed to read the actual feedback, understand the process which generated it, or engage in a dialogue about it:

Have you ever heard of someone getting “feedback” on a paper through a privileged and confidential document to HR?

A week before you go out on vacation, you see a meeting pop up at 4:30pm PST on your calendar (this popped up at around 2pm). No one would tell you what the meeting was about in advance. Then in that meeting your manager’s manager tells you “it has been decided” that you need to retract this paper by next week...

And you are told after a while, that your manager can read you a privileged and confidential document and you’re not supposed to even know who contributed to this document, who wrote this feedback, what process was followed or anything. You write a detailed document discussing whatever pieces of feedback you can find, asking for questions and clarifications, and it is completely ignored.

(from https://www.platformer.news/p/the-withering-email-that-got-a...)

I've been through the peer review process at Physical Review Letters, SIGMOD, and VLDB. You get a document containing all the reviewer's comments, plus a metareviewer's take on the overall decision and what has to change. You can engage in dialogue with the metareviewer, including a detailed response letter justifying your choices, highlighting things the reviewers may have missed, and explaining where you plan to make changes. You get additional rounds of comments from the reviewers in light of that letter on later drafts.

I'm not a Googler, and I have no idea what the standard review process looks like there, but what Gebru describes does not sound at all like peer review. I also note that Dean does not contradict Gebru's account of the meeting or feedback process. If I had a paper rejected in this fashion, I would also demand to know what the hell was going on and who was responsible.

This feels off.

Re: About Google's approach to research publication – Jeff Dean

#112

Post by bnj »

that_guy_iain wrote: post #47
bnj wrote: post #5

While I’m a spectator to this unfolding story and am reacting to a pretty cursory overview of what happened, this has the distinct feeling that Google thought they were moving in for a checkmate by, in their words, “accepting a resignation”, only to have it — very predictably? - blow up in their faces completely. Even if Google is acting presenting the chain of events faithfully, their final move to jump on an opport…

> While I’m a spectator to this unfolding story and am reacting to a pretty cursory overview of what happened, this has the distinct feeling that Google thought they were moving in for a checkmate by, in their words, “accepting a resignation”, only to have it — very predictably? - blow up in their faces completely. I wouldn't say this has blown up in their face. But to be honest, I've not seen that much drama about i…

I don’t think you’re incorrect on any particular point.

Just anecdotally, I don’t think a man sending a “do x or I’ll quit” email would be met with the same response. I think issues of race and gender are playing a role here, and that’s a shame, because while firing (accepting this resignation) of this researcher might make short term business sense, it seems like it does harm to Google’s long term credibility in trying to engage with reducing the barriers faced by (among others) women and people of color.

It’s a shame there wasn’t a pathway to keeping the dialogue going, for example by getting in touch with her and letting her know that some things she was asking for wouldn’t work, but that you could work with her to accomplish her goals in some other way.

From my privileged outside point of view, that seems like it could have been a more humane response which could have disarmed the conflict. I think opportunities to pursue options like that are de-emphasized when management are put in an all or nothing position.

A missed opportunity for leadership.

Re: About Google's approach to research publication – Jeff Dean

#113

Post by AnimalMuppet »

bnj wrote: post #41
throawaydmg wrote: post #24

> Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback It always amazes how blatantly authoritarian these "woke" types are. I would not be surprised if the sole reason she wanted the identities of…

Isn’t it pretty normal to ask to “face your accuser” and engage with feedback specifically, instead of an anonymous and unassailable red stamp?

It is. It's not a request you're ever going to get when you're dealing with the HR department, though.

Which puts you in a no-win situation when you suspect that HR is doing shady stuff against you...

Re: About Google's approach to research publication – Jeff Dean

#114

Post by ghaff »

asfadfdsa wrote: post #46

There are a lot of people commenting that she didn't actually resign. I agree, but it sounds like the conversation went like this: employee: I'm not happy about x, y and z. If you don't do those, I'm going to quit. manager: well we are not going to do those, so thank you for your time. We accept your resignation and would like it to start immediately (i.e. you're fired). If you are gonna tell your manager that you pl…

Yes. I probably didn't use the word "resign" as such but there was a time in my career when I went to my manager's manager or maybe another level up the chain and basically said I could not work for my direct manager. I got my way even though it involved working in a bit of a backwater for a time. But I was prepared to leave if I didn't get my way. (Didn't hurt that I knew said manager was a bit on the outs with the exec who I knew quite well.)

Re: About Google's approach to research publication – Jeff Dean

#115

Post by that_guy_iain »

sillysaurusx wrote: post #101
that_guy_iain wrote: post #47

Earlier quoted context omitted.

> While I’m a spectator to this unfolding story and am reacting to a pretty cursory overview of what happened, this has the distinct feeling that Google thought they were moving in for a checkmate by, in their words, “accepting a resignation”, only to have it — very predictably? - blow up in their faces completely. I wouldn't say this has blown up in their face. But to be honest, I've not seen that much drama about i…

I believe she basically sent an email to fellow employees basically telling them to stop working. That alone is a firable. That is completely nuts and unacceptable work place behaviour. The fact Google didn't fire her for that should speak volumes. This is little more than gossip unless you can point to something specific that people can judge for themselves. Saying “she basically did X” is a clever way to influence…

> This is little more than gossip unless you can point to something specific that people can judge for themselves. Saying “she basically did X” is a clever way to influence how people feel about her.

It's gossip as in I haven't seen anyone confirm the validtiy of https://www.platformer.news/p/the-withering-email-that-got-a... but I've also not seen anyone deny it. In fact, it seems everyone is confirming it unoffically.

From the blog post:

> "What I want to say is stop writing your documents because it doesn’t make a difference."

Is telling your fellow employees to stop working, if their job is writing those documents which I highly suspect it will be.

> Whether she was a good employee is kind of beside the point now.

I think it goes to show the good faith Google tried to have with someone who in my opinion was creating a hostile work envoirment.

Re: About Google's approach to research publication – Jeff Dean

#116

Post by cavisne »

gcr wrote: post #2

Maybe different teams are different, but on my previous team within Google AI, we thought the goal of google's pubapproval process was to ensure that internal company IP (eg. details about datasets, details about google compute infra) does not leak to the public, and maybe to shield Google from liability. Nothing more. In all of my time at Google AI, I never heard of pubapproval being used for peer review or to criti…

We know large language models are super important to google, and there are lots of competitors.

If they approved the paper the message would be "google thinks language models are a waste of resources and racist". There would be no academic debate on this topic as its been framed as woke and published by a militant activist, so any disagreement would be racist (see prior interactions between this researcher and other researchers [1]).

Thats why the standard process of publishing, peer reviewing, academic critique etc would not work.

Why would their researchers working on language models stay? when they can go to Facebook, OpenAI etc. Why would new researchers join?

[1] https://syncedreview.com/2020/06/30/yann-lecun-quits-twitter...

Re: About Google's approach to research publication – Jeff Dean

#117

Post by mstg »

bgribble wrote: post #79
nova22033 wrote: post #19

Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resig…

According to her Twitter feed, Timnit was asking for the same transparency that normally goes with Google pub reviews -- where all the reviewers and their feedback are known to the reviewee. This red-stamp "review" did not follow the normal review process. Not a Googler, just going on what she says: https://twitter.com/timnitGebru/status/1334881120920498180

Considering her previous attacks to anyone who disagrees with her and her previous public feuds, I'd say people may be afraid of being stamped as a racist/sexist because they had to reject her paper. She was a disruptive employee who threatened her employer and the employer didn't accept it. Why is Google obliged to keep her employed?

Re: About Google's approach to research publication – Jeff Dean

#118

Post by x0x0 »

ketzo wrote: post #85
petulla wrote: post #52

Earlier quoted context omitted.

This response really seems like gaslighting. He doesn't address her concerns and glosses over whether she was held to a different standard than others at GR.

Right -- he once again talks about "accepting" her resignation, when it reeeeeally just looks like they fired her. At the very least, she certainly feels like she was fired; why is that not mentioned at all? Even just, I don't know, "sorry we were abrupt?"

Dean previously said this [1]

> Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.

I find it unlikely Dean would lie about that, not least because the email would be easy to find.

Now, were the actions leading up to that effectively a firing, ie Timnit would have been unable to effectively continue in her role? Quite possibly.

[1] https://www.platformer.news/p/the-withering-email-that-got-a...

Re: About Google's approach to research publication – Jeff Dean

#119

Post by UncleMeat »

visarga wrote: post #108
theobeers wrote: post #33

Earlier quoted context omitted.

And the examples of issues flagged in review that Jeff keeps highlighting—like Timnit’s alleged failure to mention recent work to reduce the environmental impact of large models—are themselves a bit worrisome. Jeff gives the impression that they demanded retraction (!) because they wanted Timnit and her coauthors to soften their critique. The more I read about this, the worse it looks.

> Jeff gives the impression that they demanded retraction (!) In paper reviews you can often see reviewers asking the authors to rewrite, clarify, add extra experiments, add missing citations. It's all normal.

Usually those are "accept with minor revisions" or "revise and resubmit". Rarely are they grounds for complete rejection. This is extra true for internal review, since the actual conference review process would provide an additional layer to ensure that the scholarship was strong.

Re: About Google's approach to research publication – Jeff Dean

#120

Post by nebulous1 »

ghaff wrote: post #77
wilde wrote: post #50

Earlier quoted context omitted.

Not really. People bluff in negotiations all the time. A normal response would have been, “I can’t do that. Are you still committed to leaving?” Instead Google went all scorched earth and it’s still unclear why. It also indicates that they don’t particularly value her in progress work in the lab since normally you want the 2 weeks or whatever to squeeze some wrap up work out of folks.

People bluff but you generally have to be prepared to walk if your bluff is called. If I walk into a manager's office and demand 2x salary or I walk and the response is "we believe you are fairly compensated," you're now in a rather uncomfortable situation. Which is of course not to say you can't have a more measured negotiation. But it can be hard to walk back from give me X or I do Y, especially if there isn't a lo…

> People bluff but you generally have to be prepared to walk if your bluff is called.

Then you aren't bluffing.

Post reply on HN