Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jeff Dean responds to EDA industry about AlphaChip

twitter.com

Post reply on HN
Print view View original

11–20 of 221 posts

Re: Jeff Dean responds to EDA industry about AlphaChip

#11

Post by refulgentis »

bayarearefugee wrote: post #10
danpalmer wrote: post #6

Earlier quoted context omitted.

> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.

At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the s…

> why does anyone care

n.b. you're on a social news site

> pull way ahead of all these other "idiots"

Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong?

> "idiots"

?

Re: Jeff Dean responds to EDA industry about AlphaChip

#12

Post by skybrian »

twothreeone wrote: post #9
1024core wrote: post #5

Earlier quoted context omitted.

> they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?

It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.

I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers.

Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right.

The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.

Re: Jeff Dean responds to EDA industry about AlphaChip

#13

Post by xpe »

twothreeone wrote: post #9
1024core wrote: post #5

Earlier quoted context omitted.

> they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?

It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.

> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying.

> It's not a value judgement, just doesn't help his case at all.

Calling it "bullying" looks like a value judgment to me. Am I missing something?

To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.

Re: Jeff Dean responds to EDA industry about AlphaChip

#14

Post by make3 »

twothreeone wrote: post #2

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

"they couldn't replicate it because they're idiots, therefore it's replicable" That's literally not what he says though. He says, "they didn't replicate it so their conclusions are invalid", which is a completely different thing than what you're accusing him of, and is valid.

Re: Jeff Dean responds to EDA industry about AlphaChip

#15

Post by pclmulqdq »

bayarearefugee wrote: post #10
danpalmer wrote: post #6

Earlier quoted context omitted.

> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.

At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the s…

The reason Jeff Dean cares is that his team's improvement compared to standard EDA tools was marginal at best and may have overfitted to a certain class of chips. Thus, he is defending his research because it is not widely accepted. Open source code has been out for years and in that time the EDA companies have largely done their own ML-based approaches that do not match his. He attributes this not to failings in his own research but to the detractors at these companies not giving it a fair chance.

The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked).

The fact that the result is so minor is the reason that this is so contentious.

Re: Jeff Dean responds to EDA industry about AlphaChip

#16

Post by twothreeone »

jsnell wrote: post #8
twothreeone wrote: post #2

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

It's a good thing that he didn't say that, then. The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.

A Google Nature Paper has not been replicated for over 3 years, but I'm the one fabricating stuff :D

Making a novel claim implies its *_claimed_ replicability.

"You did not follow the steps" is calling them idiots.

The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.

Re: Jeff Dean responds to EDA industry about AlphaChip

#17

Post by xpe »

bsder wrote: post #4

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…

> Google continues to peddle unsubstantiated snake oil

I read your comment, but I'm not following -- or maybe I disagree with it -- I'm not sure yet.

"Snake oil" is an emotionally loaded term that raises the temperature of the conversation. That usually makes having a conversation harder.

From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. Agree? Are you claiming that AlphaChip is not? Are you claiming they are perpetrating some kind of deception or exaggeration? Your numbered points seem like valid criticisms (I haven't evaluated them closely), but even if true, I don't see how they support your "snake oil" claim.

Re: Jeff Dean responds to EDA industry about AlphaChip

#18

Post by twothreeone »

skybrian wrote: post #12
twothreeone wrote: post #9

Earlier quoted context omitted.

It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.

I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers. Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right. The only way to avoid that is to get critical feedback before publishing the paper, and…

The situation here is different though.. If I'm making an existence claim by demonstrating a constructive argument and then being criticized for it, the most effective response to that critique would be a second, alternative construction, not attacking the critic's argument. After all, I'm the one claiming existence.. the burden of proof is on me, not my critics.

Re: Jeff Dean responds to EDA industry about AlphaChip

#19

Post by xpe »

bsder wrote: post #4

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…

> if it worked, they wouldn't be sharing it and would simply be printing money by taking advantage of it.

Sure, there are some techniques in financial markets that are only valuable when they are not widely known. But claiming this pattern applies universally is incorrect.

Publishing a technique doesn't prove it doesn't work. (Stating it this way makes it fairly obvious.)

DeepMind, like many AI research labs, publish important and useful research. One might ask "is a lab leaving money off the table by publishing?". Perhaps a better question is "What 'game' is the lab playing and over what time scale?".

Re: Jeff Dean responds to EDA industry about AlphaChip

#20

Post by griomnib »

twothreeone wrote: post #2

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

I love how he’s claiming bias due to his critic’s employer. As though working for Google has no conflicts? A company that is desperately hyping each and every “me too” AI development to juice the stock price?

Jeff drank so much kool aid he forget what water is.

Post reply on HN