Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

About Google's approach to research publication – Jeff Dean

docs.google.com

Post reply on HN
Print view View original

311–320 of 607 posts

Re: About Google's approach to research publication – Jeff Dean

#311

Post by geofft »

tooltalk wrote: post #270
whimsicalism wrote: post #193

Earlier quoted context omitted.

> academic journal peer review process Fro my understanding, this paper had already passed peer review and been accepted. Google management then decided to block the publication using the IP review process.

Please go read the link first. Jeff clearly states that Google has a review protocol for journal submission which requires a two weeks internal review period. Timnit shared the paper a day before the publication deadline, ie, no time for internal review, and someone with a fat finger apparently approved it for submission without the required review.

That's not under dispute. What's under dispute is:

1) Is the review protocol that requires a two-week review period a peer review process intended to maintain scientific rigor, or an internal controls process intended to prevent unwanted disclosure of trade secrets, PII, etc.?

Repeating the comment at the very top of the thread:

> Maybe different teams are different, but on my previous team within Google AI, we thought the goal of google's pubapproval process was to ensure that internal company IP (eg. details about datasets, details about google compute infra) does not leak to the public, and maybe to shield Google from liability. Nothing more.

If it's not a scientific peer review process, arguments about why scientific peer review is generally anonymous are irrelevant, just like arguments about why, say, code review is generally not anonymous would also be irrelevant. It's a different kind of review process from both of those.

2) In practice, is the two-week review period actually expected / enforced? Other Googlers, including people in her organization, are saying that the two week requirement is a guideline, not a hard rule, and submissions on short notice are regularly accepted without complaint:

https://twitter.com/le_roux_nicolas/status/13346245318860718...

https://twitter.com/ItsNeuronal/status/1334636596113510400

https://twitter.com/lizthegrey/status/1334659334689570817

(I don't work for Google, but I work for another very IP-leak-sensitive employer that does ML stuff, and we have a two-week review period on publications. The two-week rule exists for the purpose of not causing last-minute work for people, but if you miss it, it's totally permissible to bug folks to get it approved, and if they do, it's not considered "someone with a fat finger." It certainly doesn't exist for the purpose of peer review - it's assumed that the venue you're submitting to will do review, and I think everyone understands that someone from your own employer isn't going to be a fair peer reviewer anyway. There is a "technical reviewer" of your choice, but basically they just make sure you're not embarrassing yourself and the company, and there's no requirement for how deeply they review. I think I've gone through the process twice and missed the deadlines both times.)

So, if this "rule" exists on paper, but only exists in practice for her, then this is the textbook definition of unfairness.

Re: About Google's approach to research publication – Jeff Dean

#312

Post by rebelos »

apta wrote: post #208
tinyhouse wrote: post #191

In her email she claims that she is constantly being dehumanized. This is unacceptable to someone in her amazing position and to be honest sounds like she is a narcissist. I think Google couldn't wait to get rid of her (cannot blame them, she wanted to sue the company a year prior, also "represented" the company really bad on social media) and her email with demands was their opportunity. I don't think she would surv…

This is the problem with hiring SJWs. They will claim being victimized, racism, etc. the moment they don't have their way. I'm sure many people would have loved to be in her position and have her salary and benefits. Some people can't have enough it seems. "Wokeness" is a disease, the goal is to keep getting more and more power behind the guise of "equality". Hopefully more employers will reject hiring "woke" people…

I suspect you're being downvoted because this post sounds like a denial of very real issues of sexism and racism.

However, the suggestion that employers will work to identify and preemptively reject employees with an SJW bent is not only sound but almost certainly already underway.

Re: About Google's approach to research publication – Jeff Dean

#313

Post by aabhay »

eachro wrote: post #107
gcr wrote: post #2

Maybe different teams are different, but on my previous team within Google AI, we thought the goal of google's pubapproval process was to ensure that internal company IP (eg. details about datasets, details about google compute infra) does not leak to the public, and maybe to shield Google from liability. Nothing more. In all of my time at Google AI, I never heard of pubapproval being used for peer review or to criti…

But in this scenario, shielding Google from liability is actually a primary concern given that Timnit is discussing ethics/bias. A paper on say novel transformer architecture, the lottery ticket hypothesis in a new setting, a new RL benchmark suite, etc is not going to expose Google to legal risk the way ethical AI research often can.

This. I have been arguing the unpopular opinion that most AI ethics work in corporate settings is not designed to empower real research. It was a matter of time before an actual researcher with an ethical compass was removed unceremoniously. Anyone in an AI ethics team at a large company — you need to know exactly what your job means to the company, because it isn’t safe.

Re: About Google's approach to research publication – Jeff Dean

#314

Post by geofft »

that_guy_iain wrote: post #67
theobeers wrote: post #33

Earlier quoted context omitted.

And the examples of issues flagged in review that Jeff keeps highlighting—like Timnit’s alleged failure to mention recent work to reduce the environmental impact of large models—are themselves a bit worrisome. Jeff gives the impression that they demanded retraction (!) because they wanted Timnit and her coauthors to soften their critique. The more I read about this, the worse it looks.

> Jeff gives the impression that they demanded retraction (!) because they wanted Timnit and her coauthors to soften their critique. The more I read about this, the worse it looks. I get the impression that she wrote a hit piece on Google and published with Google's name. For me, it's correct they demand a retraction. It's simply unprofessional to critque your company for something while not mentioning the work they'…

For me, it's pretty unprofessional (and cowardly) to hire a well-respected ethics researcher, write some PR pieces about how the company takes ethical actions seriously, and then tell her that her publications have to follow the party line and cannot overly criticize the company.

Re: About Google's approach to research publication – Jeff Dean

#315

Post by nickff »

neltnerb wrote: post #294
nickff wrote: post #224

Earlier quoted context omitted.

As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.

Uh, or you could compromise with them and act like two adults? This isn't a zero sum game.

If someone wants to negotiate, they don't use an ultimatum.

Re: About Google's approach to research publication – Jeff Dean

#316

Post by ryanobjc »

SpicyLemonZest wrote: post #102
reikonomusha wrote: post #94

Earlier quoted context omitted.

What if the language weren’t so direct? For example, > If I can’t know who is providing reviews to my work, it’s difficult for me to imagine continuing to work as a part of this group. Many people would still describe this (erroneously) as an ultimatum even though it could be bleakly summarized in the same vein as a true ultimatum.

We don't know the exact language, but by Timnit's account she made a numbered list of demands, which is pretty inherently direct.

And she said her email contained language like “and if my demands aren’t met, then I will find an exit date.”

It sounded from her account a straightforward threat to leave.

Re: About Google's approach to research publication – Jeff Dean

#317

Post by nostrademons »

neltnerb wrote: post #294
nickff wrote: post #224

Earlier quoted context omitted.

As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.

Uh, or you could compromise with them and act like two adults? This isn't a zero sum game.

Isn't the definition of "ultimatum" that no further compromise is possible? You can try or offer different alternatives, but if the other person is really at the ultimatum stage then you've both already lost.

Re: About Google's approach to research publication – Jeff Dean

#318

Post by alangibson »

nickff wrote: post #224
UncleMeat wrote: post #109

Earlier quoted context omitted.

If your work is suddenly and unexpected roadblocked at the last second by internal review, the only way to make changes that prevent that from happening in the future is to clearly understand the situations and criticisms that led to the roadblock. This is why understanding who raised these concerns is important. Anonymous feedback blowing up a project at the final moments is sure to frustrate anybody. If what she ha…

As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.

That is a psychotic reaction that assumes the world is just a Hobbesian nightmare of domination or death.

Re: About Google's approach to research publication – Jeff Dean

#319

Post by pgodzin »

UncleMeat wrote: post #215
siliconmountain wrote: post #149

Earlier quoted context omitted.

Isn't the academic journal peer review process generally an anonymous feedback mechanism? Why does this need to be different?

This isn't (or wasn't) a review process for scholarship. Oodles of people within Google (even within Brain) have gone through this process and it seems to have always been the case that it just checks for things like PR problems, IP leaks, etc. Further, she claims that initially she was not allowed to even see the contents of the criticisms, only that the paper needed to be withdrawn. Let's say you were working on a…

Seems like part of this was that the paper represented a PR problem

Re: About Google's approach to research publication – Jeff Dean

#320

Post by tempest_ »

geodel wrote: post #16
hartator wrote: post #9

It sadden me that they make Jeff Dean work on these kind of issues. His map/reduce invention brought to the world a lot more than managing his coworker egos.

How do you know they are making him to this. It seems to me Jeff chose to accept leadership position at Google. So managing egos would be first class task to deal with for him.

Exactly.

If managing people were the same skill set as computer science perhaps we would not be seeing this play out as it has.

This is seriously a massive failure on the part of Google to handle their own shit. I often wonder if these people are too old or out of touch to appreciate how this kind of thing may or may not blow up on social media.

To not consider how this might cause public relations issues especially when dealing with someone who has an established social media presence seems like a misstep.

On the flip side maybe they thought it was worth it. Weather the storm for a few days and twitter will move one. It's hard to say.

Post reply on HN