About Google's approach to research publication – Jeff Dean • Hacker News Board

Re: About Google's approach to research publication – Jeff Dean

#551

Post by bnj » Sat, Dec 05, 2020, 4:54 AM UTC

sidlls wrote: post #83

bnj wrote: post #41

Earlier quoted context omitted.

Isn’t it pretty normal to ask to “face your accuser” and engage with feedback specifically, instead of an anonymous and unassailable red stamp?

In the context of peer review it’s common, or at least not uncommon, for reviewers to be anonymous. Personally I think it should be completely blind both directions to promote maximum objectivity. Whether that applies at all in this case is a separate matter.

This is a great point I hadn’t considered the academic peer review component, to whatever extent that plays a role with the internal process they use

Re: About Google's approach to research publication – Jeff Dean

#552

Post by tinyhouse » Sat, Dec 05, 2020, 4:57 AM UTC

I read the abstract. The paper doesn't critique Google. It critiques the current focus of AI research of building bigger and bigger models. Google is one of the leaders in building those big models. So what? They also invest in research to make smaller models that perform equally well. They have a huge incentive - those big models are very expensive. I don't think anyone at Google really care if the paper is published. I'm pretty sure it will get published sooner or later. The authors could've addressed the reviewers' comments pretty quickly and then re-submit to another conference. The whole thing blew out of proportion. She picked the wrong battle. Should've bitten the bullet.

Re: About Google's approach to research publication – Jeff Dean

#553

Post by ninjin » Sat, Dec 05, 2020, 5:05 AM UTC

htrp wrote: post #493

omgwtfbyobbq wrote: post #59

Earlier quoted context omitted.

That was my impression as well. Having said that, if Jeff were to make public the paper, criticisms of the paper, and improvements made to address the problems described in the paper, that could go a long way towards clearing the air.

Abstract from a reviewer (source reddit /r/ml) Abstract The past three years of work in natural language processing have been characterized by the development and deployment of ever larger language models, especially for English. GPT-2, GPT-3, BERT and its variants have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pre- trained models and the methodol…

Link to the Reddit thread:

https://old.reddit.com/r/MachineLearning/comments/k69eq0/n_t...

I have plenty of experience in Natural Language Processing (NLP), but I am not an expert in ethics and bias – although I have read my fair share of papers on it. To me, the abstract comes across as modest, very reasonable, and exploring questions highly relevant to the community as a whole. Sure, if I was to review it I would be picky in regards to any strong empirical claims since I suspect it would be difficult to demonstrate conclusively some aspects of what they hint towards in the abstract. But as a position paper it looks better than plenty of work already published at top-tier NLP venues and I doubt that it could not get accepted on academic merits.

Still, to echo the parent, does anyone have the paper in its entirety?

Re: About Google's approach to research publication – Jeff Dean

#554

Post by threwo324324 » Sat, Dec 05, 2020, 5:08 AM UTC

htrp wrote: post #199

UncleMeat wrote: post #109

Earlier quoted context omitted.

If your work is suddenly and unexpected roadblocked at the last second by internal review, the only way to make changes that prevent that from happening in the future is to clearly understand the situations and criticisms that led to the roadblock. This is why understanding who raised these concerns is important. Anonymous feedback blowing up a project at the final moments is sure to frustrate anybody. If what she ha…

>Now might be a good time to remind everyone that the easiest way to discriminate is to make stringent rules, then to decide when and for whom to enforce them. --Nicolas Le Roux

This is how most 'fascist' states work- theoretical controls on each and every aspect, laxly imposed on everyone but the deplorables.

Re: About Google's approach to research publication – Jeff Dean

#555

Post by cbsmith » Sat, Dec 05, 2020, 5:10 AM UTC

nickff wrote: post #224

UncleMeat wrote: post #109

Earlier quoted context omitted.

If your work is suddenly and unexpected roadblocked at the last second by internal review, the only way to make changes that prevent that from happening in the future is to clearly understand the situations and criticisms that led to the roadblock. This is why understanding who raised these concerns is important. Anonymous feedback blowing up a project at the final moments is sure to frustrate anybody. If what she ha…

As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.

I'm sorry, but that's no more true than when the ultimatum is the other way around.

I think that statement presumes some degree of unreasonableness. Honestly, I value having employees that have principles and clear boundaries, if for no other reason than I can rest assured that when I'm not observing/involved, those principles and clear boundaries are still there. Now, if those principles are, "I won't accept that paying me gives you any kind of authority over what I do", then you know that's not going to work out for anyone involved. However, if it is something like, "You can't pay me enough to do X", and I have no desire for them to do X, I'm really okay with that.

Re: About Google's approach to research publication – Jeff Dean

#556

Post by aaplok » Sat, Dec 05, 2020, 5:19 AM UTC

kyrieeschaton wrote: post #522

Ar-Curunir wrote: post #489

Earlier quoted context omitted.

"based on her track record"? The fuck does that mean? If you wanna say that people shouldn't call out their employers, then say that. No need for character assassination.

Anyone can look up her tweets and her interactions with, eg, Yann Lecun, and make a pretty good inference.

I looked them up. I found the exchange robust but not extraordinarily so. She was firm but didn't seem to step out of line.

What inference was I supposed to make?

Re: About Google's approach to research publication – Jeff Dean

#557

Post by joshuamorton » Sat, Dec 05, 2020, 5:27 AM UTC

refulgentis wrote: post #538

joshuamorton wrote: post #308

Earlier quoted context omitted.

Jeff's statement is a lie. I work at Google, and the publication approval process doesn't normally take to weeks. Any Google employee can verify this, and others have, both in this thread and on twitter. There's some documentation that asks for 2 weeks, but it's not followed, and inconsistent application of policy is absolutely a concern in a case like this. A summary of the events, as best as I can tell, is the norm…

I work at Google too, and it's embarrassing to see you attack your colleagues like this in public - how is Jeff's statement "a lie"? Whether or not you believe the 2 week deadline is a hard and fast rule followed 100% of the time, Jeff never made that claim. 1 day is certainly unreasonable. What makes you think they weren't given the opportunity? We were told specifically they _were_ given an opportunity to revise, a…

> Whether or not you believe the 2 week deadline is a hard and fast rule followed 100% of the time, Jeff never made that claim

His words were, and I quote "we require two weeks for this sort of review". That is an absolutely false statement, and Jeff knows it. I see no other characterization than a lie. I take no solace in that fact, priors were that Jeff Dean was an above average executive and ran a more ethical then average org. I'm disappointed to see my trust was misplaced and that Jeff was willing to endorse that document, but I'm not going to not hold someone accountable when I see them do something obviously wrong, even if we work for the same employer. Ethics extend beyond that.

> 1 day is certainly unreasonable.

It is not though. It's the norm.

> We were told specifically they _were_ given an opportunity to revise

This is not true (if you believe this to be true, can you cite said statement?). I'd be more than happy to discuss this with you privately, my username should be obvious ;)

Re: About Google's approach to research publication – Jeff Dean

#558

Post by cbsmith » Sat, Dec 05, 2020, 5:29 AM UTC

saagarjha wrote: post #401

nickff wrote: post #376

Earlier quoted context omitted.

Well, the company is in slightly different position from the manager. They can abrogate the manager's authority, but that would permanently undermine that individual. On the other hand, they can also choose to accept the subordinate's resignation. They could try to transfer the subordinate somewhere else, but that's also risky, and wouldn't really address the ultimatum in this case. Accepting a resignation achieves t…

I think you're coming at the ultimatum from some sort of strange power dynamic perspective, where an employee who successfully gets their ultimatum approved somehow disenfranchises their manager of their authority, enabling future employees to…what? Vie for the managerial position? This has a "crush dissent" kind of vibe. In every employment contract there is a balance of things and employee is willing to do and an e…

> This has a "crush dissent" kind of vibe.

Yeah, and whether intended or not, a "fire anyone who gives you an ultimatum" strategy absolutely creates that vibe.

If you have a top down management style where you employees do not question anything you say, that might be the way to go, but I find in the software business what you want is the opposite. You want all the criticism and feedback you can get from your skilled and knowledgeable work force. If you don't get that, you're wasting the majority of that money in their pay check.

The irony here is that if you have a manager firing someone who presents an ultimatum, then tat in itself is effectively an ultimatum that you are supporting. ;-)

That of course also doesn't mean you accede to every ultimatum. I mean, if your business plan is to do X, you want employees that will help you to do X. If they are getting in the way of X, then you need different employees anyway. Usually though, you and they have already worked out that they want to work with you to help you do X before you hire them.

So the main reason you get ultimatums is because they didn't anticipate and do not like the approach you are taking to get to X. Assuming they are smart and have good judgement (and again, if not, why did you hire them? why are you paying them?), there's a very good chance that there are some problems with your approach and you'd be wise to at least consider that possibility and their perspective. They may be trying to save you from making a terrible mistake, and feel like it is incumbent on them to stop working for you because allowing you to proceed would be working against that goal you hired them for.

It's not uncommon for two people to have very different perspectives on what helps to achieve a company's objective. It's also not uncommon for one of those people to be horribly, horribly wrong. Sure, if you've got an employee who has presented an ultimatum based on horribly wrong judgement, it may make no sense be their employer.

I'll tell you though... just because their a subordinate doesn't automatically mean they are the ones exercising horrible judgement... and the farther you go up the food chain, the more severe the consequences from supporting someone's horrible misjudgement. So having a policy of summarily firing subordinates who present ultimatums both creates the wrong environment to get the best out of your team and terribly harmful for the leadership of your organization.

Re: About Google's approach to research publication – Jeff Dean

#559

Post by jkaplowitz » Sat, Dec 05, 2020, 5:32 AM UTC

nickff wrote: post #224

UncleMeat wrote: post #109

Earlier quoted context omitted.

If your work is suddenly and unexpected roadblocked at the last second by internal review, the only way to make changes that prevent that from happening in the future is to clearly understand the situations and criticisms that led to the roadblock. This is why understanding who raised these concerns is important. Anonymous feedback blowing up a project at the final moments is sure to frustrate anybody. If what she ha…

As a manager, if someone gives an ultimatum, you basically have to fire them. There's no real option; Ben Horowitz covered it somewhere, but the bottom line is that if you yield, you've given up all control.

I don't want to work for a manager who will never think: "Huh, this situation is serious enough for someone to make this kind of ultimatum. Who is right and why? Let me take a moment to think about it with an open mind and pick the most appropriate reaction regardless of what I previously thought."

Sometimes the person making an ultimatum is right, sometimes they're wrong. It shouldn't be as adversarial as viewing yielding as weak. Insisting on always "winning" is in my view the weak position.

Additionally, firing someone is not always legal in some countries, even after an ultimatum, assuming they pick the wording of their ultimatum carefully (e.g. "I may very well resign if/unless [desired condition]") to retain control over whether they will later finalize their conditional decision to resign.

As one example, in Quebec, employees who don't qualify as "senior management" and who have been employed at a company there for an uninterrupted period of at least 2 years cannot legally be fired without what the law considers good cause, period, not even if the company gives them a notice period or pay in lieu. Any alleged noncompliance or misconduct that falls short of the most extreme examples must be first dealt with a graduated process of progressively stronger discipline, and it must be possible for someone to recover from that instead of having the outcome of the process as a foregone conclusion. There is a government tribunal to which an aggrieved party can appeal if they aren't happy with the outcome, with the power to order remedies including back pay and even reinstatement.

Similar things are found in many European countries, though certainly not all.

Of course, ultimatums with more definitive wording like "I resign if/unless [condition that the listener has control over]" -- note the absence of hesitating words like "may very well" -- can irreversibly become an effective resignation worldwide, based on choice of the listener on whether to satisfy or reject the condition.

Re: About Google's approach to research publication – Jeff Dean

#560

Post by cbsmith » Sat, Dec 05, 2020, 5:41 AM UTC

freeone3000 wrote: post #398

acituan wrote: post #369

Earlier quoted context omitted.

Ethics researcher omits relevant research in paper to smuggle in a particular agenda. Company calls it out. Researcher realizes the limits of her narrative setting powers. Non-vocal majority is happy to see runaway activism being curbed in corporate settings. Slow march through institutions is slowed down for a day.

It's not "runaway activism" if you hire ethics researchers who find ethics problems. Reasons for not citing research, especially recent research, range from lack of relevance (since even though environmental improvements could have been done, they were not done! So the actual impact wasnt lessened by them at all!); To simply not having known about it. The correct reviewer response to this would be an "accept with cor…

> It's not "runaway activism" if you hire ethics researchers who find ethics problems.

Agreed, but there's obviously a difference of perspective about what transpired here and I don't think any of us knows with certainty what the truth is. Finding ethics problems is grand and all, but framing a narrative that misrepresents those problems is highly problematic, particularly if they are in a role that makes them the leading ethical voice of the company.