Earlier quoted context omitted.
oh yes absolutely, i would love for timmit or jeff to release this paper as-is so we can see what happened. could be a good read.
The draft is rolling around the web already.
link?
About Google's approach to research publication – Jeff Dean
291–300 of 607 posts
Re: About Google's approach to research publication – Jeff Dean
#292After reading Jeff Dean's response, I can only come to the conclusion that Timnit Gebru acted like a primadonna. It is completely normal to have to obtain prepublication signoff on material before it is submitted to a conference (and manager signoff even before abstract submission). Given the breadth of experience at Google, it seems strange not to avail yourself of this. Demanding to know the identity of reviewers i…
> It is completely normal to have to obtain prepublication signoff on material before it is submitted to a conference (and manager signoff even before abstract submission) To be clear, both of these had been received for the paper in question.
That isn't the impression I had from reading the principal's statements, to wit:
> Unfortunately, this particular paper was only shared with a day’s notice before its deadline — we require two weeks for this sort of review — and then instead of awaiting reviewer feedback, it was approved for submission and submitted.
It seems like someone short-circuited the review process and submitted without review to meet a deadline, with a post-submission review. When this occurred, it was not all green lights. The expectation is that you then pull the submission and, post review, submit elsewhere. After all, conferences are all virtual now, it is not a hardship to submit elsewhere.
If you have other information as to the facts or the internal process do tell.
Re: About Google's approach to research publication – Jeff Dean
#293Earlier quoted context omitted.
Unless she lied in her first e-mail, which it doesn't seem like she did, the reason she made those demands is because they asked for a retraction of the paper without indicating why the paper should be retracted. Asking for the identity of people that have the authority to ask for a withdrawal of your research without stating their issues with it seems understandable, if excessive. But maybe I misunderstood something…
AIUI, they asked her to retract it because she submitted it before getting final approval, and then they in fact decided not to approve it.
Dean's statement is clear that it was approved before being submitted:
> Unfortunately, this particular paper was only shared with a day’s notice before its deadline — we require two weeks for this sort of review — and then instead of awaiting reviewer feedback, it was approved for submission and submitted. A cross functional team then reviewed the paper as part of our regular process and the authors were informed that it didn’t meet our bar for publication and were given feedback about why. [...] We acknowledge that the authors were extremely disappointed with the decision that Megan and I ultimately made, especially as they’d already submitted the paper.
There is no statement at all of how to reconcile "approved for submission" with "didn’t meet our bar for publication", which probably means that there is no reconciliation, and the cancellation was done outside normal process.
Re: About Google's approach to research publication – Jeff Dean
#294Earlier quoted context omitted.
If your work is suddenly and unexpected roadblocked at the last second by internal review, the only way to make changes that prevent that from happening in the future is to clearly understand the situations and criticisms that led to the roadblock. This is why understanding who raised these concerns is important. Anonymous feedback blowing up a project at the final moments is sure to frustrate anybody. If what she ha…
As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.
Uh, or you could compromise with them and act like two adults? This isn't a zero sum game.
Re: About Google's approach to research publication – Jeff Dean
#295I think we’re starting to see companies discover the limits of how far they’re willing to let employees push their “woke” agenda using the company’s name. It seems disparaging your own company while ignoring research that counters yours is Google’s limit, but we’ll have to wait to see the research paper if it leaks.
Reading the abstract, it seems to be a pretty vanilla-esque survey paper.... which implies there's some more info that will only come out if this goes to court (doubtful).
Re: About Google's approach to research publication – Jeff Dean
#296Maybe different teams are different, but on my previous team within Google AI, we thought the goal of google's pubapproval process was to ensure that internal company IP (eg. details about datasets, details about google compute infra) does not leak to the public, and maybe to shield Google from liability. Nothing more. In all of my time at Google AI, I never heard of pubapproval being used for peer review or to criti…
We know large language models are super important to google, and there are lots of competitors. If they approved the paper the message would be "google thinks language models are a waste of resources and racist". There would be no academic debate on this topic as its been framed as woke and published by a militant activist, so any disagreement would be racist (see prior interactions between this researcher and other…
Academic debate is, in fact, done through conferences and journals. You saying there can be no debate is a strawman position with no basis in reality. The idea that standard rigor cannot be applied to ethics research is absurd, and seems to insinuate that the entire field is absent discipline.
The proper response to her position would be to publish a response or critique. Attacking her entire field does nothing to further the conversation.
Re: About Google's approach to research publication – Jeff Dean
#297I don't know Jeff Dean. I have read some of his work, watched some of his presentations. He seems a credible bloke. This, though, looks and feels like thinly-veiled retroactive and pretty unconvincing PR. It's short on detail and appears somewhat at odds with several points from Timnit Gebru's resignation note [0]: - Dean says the paper was reviewed by a "cross-functional team". Gebru says she received the feedback t…
> Gebru says she received the feedback through a "privileged and confidential document to HR"
I agree that Google isn't looking good at the moment. But if I had a colleague who seemed intent on finding ways to place the company in legal jeopardy, then I too would avoid direct communication whenever possible.
Re: About Google's approach to research publication – Jeff Dean
#298Earlier quoted context omitted.
> ...if we didn’t meet these demands, she would leave Google and work on an end date. To be fair it doesn't sound like her ultimatum was "I'll leave immediately"— that was forced on her by Google, and is an important detail.
Definitely. "If we don't fix this I don't think I can continue working here" is not a resignation, it's a negotiating position.
Sure, but sending out an e-mail accusing your colleagues of racism, exhorting them to stop working, and talking about potential lawsuits isn't. There's no way that Google (or any company) would continue to employ her after that.
Re: About Google's approach to research publication – Jeff Dean
#299Earlier quoted context omitted.
> People bluff but you generally have to be prepared to walk if your bluff is called. Then you aren't bluffing.
How do you figure that? If I hold 27 off suit then I still have to be prepared to show my hand if you call me. Doesn't mean my all-in raise wasn't bluffing.
Yeah, but your all-in raise is you saying that you have two aces. You're bluffing because you don't actually have the hand you're pretending to have.
Re: About Google's approach to research publication – Jeff Dean
#300Earlier quoted context omitted.
> It is completely normal to have to obtain prepublication signoff on material before it is submitted to a conference (and manager signoff even before abstract submission) To be clear, both of these had been received for the paper in question.
That isn't the impression I had from reading the principal's statements, to wit: > Unfortunately, this particular paper was only shared with a day’s notice before its deadline — we require two weeks for this sort of review — and then instead of awaiting reviewer feedback, it was approved for submission and submitted. It seems like someone short-circuited the review process and submitted without review to meet a deadl…
How do you interpret "approved for submission" as "submitted without review"?