Earlier quoted context omitted.
> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.
At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the s…
Jeff Dean responds to EDA industry about AlphaChip
71–80 of 221 posts
Re: Jeff Dean responds to EDA industry about AlphaChip
#72Earlier quoted context omitted.
And do you believe that that is what's happening in this case? If you have personal experience with Jeff Dean et al that you're willing to share, I'd be interested in hearing about it. From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss." It's not even clea…
> From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss." From where I'm sitting it looks like Google cooked the books maximally, barely beat humans let alone state of the art algorithms, published a crappy article in Nature because it would never have passed…
Likewise the importance of spending 20x as much money on the training portion seems easy to verify, and significant.
That they would fail to properly test against industry standard workbenches seems reasonable to me. This is a bunch of ML specialists who know nothing about chip design. Their background is beating everyone at Go and setting a new state of the art for protein folding, and not chip design. If you dismiss those particular past accomplishments as hyperbolic marketing, that's your decision. But you aren't going to find a lot of people in these parts who agree with you.
If you think that those were real, but that a bunch of more recent accomplishments are BS, I haven't been following closely enough to have an opinion. The stuff that crossed my radar since AlphaFold is mostly done at places like OpenAI, and not Google.
Regardless, the truth will out. And what Google is claiming for itself here really isn't all that impressive.
Re: Jeff Dean responds to EDA industry about AlphaChip
#73The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…
This is a fallacious argument. A better chip design process does not eliminate all other risks like product-market fit or the upfront cost of making masks or chronic mismanagement.
Re: Jeff Dean responds to EDA industry about AlphaChip
#74Curious why there's so much emotion and unpleasantness in this dispute? How did it evolve from the boring academic argument about benchmarks, significance, etc to a battle of personal attacks?
Re: Jeff Dean responds to EDA industry about AlphaChip
#75I have published an addendum to an article I wrote about AlphaChip ( https://vighneshiyer.com/misc/ml-for-placement/ ) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still ma…
In the conclusion of the article, you said: "While I concede that there are things the ISPD authors could have done better, their conclusion is still sound. The Nature authors do not address the fact that CMP and AutoDMP outperform CT with far less runtime and compute requirements." One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To…
I believe this is a fair criticism, and it could be a reason why the ISPD Tensorboard shows divergence during training for some RTL designs. The ISPD authors provide their own justification for their substitution of training time for compute resources in page 11 of their paper (https://arxiv.org/pdf/2302.11014).
I do not think it changes the ISPD work's conclusions however since they demonstrate that CMP and AutoDMP outperform CT wrt QoR and runtime even though they use much fewer compute resources. If more compute resources are used and CT becomes competitive wrt QoR, then it will still lag behind in runtime. Furthermore, Google has not produced evidence that AlphaChip, with their substantial compute resources, outperforms commercial placers (or even AutoDMP). In the recent rebuttal from Google (https://arxiv.org/pdf/2411.10053), the only claim on page 8 says Google VLSI engineers preferred RL over humans and commercial placers on a blind study conducted in 2020. Commercial mixed placers, if configured correctly, have become very good over the past 4 years, so perhaps another blind study is warranted.
> Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article
I will openly admit my bias is against the AlphaChip work. I referred to the Nature authors as 'arrogant' and 'disdainful' with respect to their statement that EDA CAD engineers are just being bitter ML-haters when they criticize the AlphaChip work. I referred to Jeff Dean as 'belittling' and 'hostile' and using 'hyperbole' with respect to his statements against Igor Markov, which I think is unbecoming of him. I referred to Shankar as 'excellent' with respect to his shrewd business acumen.
Re: Jeff Dean responds to EDA industry about AlphaChip
#76Earlier quoted context omitted.
>peer reviewed research published in journals Peer review doesn't mean as much as Elsevier would like you to believe. Plenty of peer-reviewed research is absolute trash.
All of the highest impact papers authored by DeepMind and Google Brain have appeared in Nature, which is the gold standard for peer-reviewed natural science research. What exactly are you trying to claim about Google's peer-reviewed papers?
In and of itself, "Being published in a peer reviewed journal" does not place the contents of a paper beyond reproach or criticism.
Re: Jeff Dean responds to EDA industry about AlphaChip
#77Earlier quoted context omitted.
In the conclusion of the article, you said: "While I concede that there are things the ISPD authors could have done better, their conclusion is still sound. The Nature authors do not address the fact that CMP and AutoDMP outperform CT with far less runtime and compute requirements." One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To…
> One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this? I believe this is a fair criticism, and it could be a reason why the ISPD Tensorboard shows divergence during training for some RTL designs. The ISPD autho…
Re: Jeff Dean responds to EDA industry about AlphaChip
#78Re: Jeff Dean responds to EDA industry about AlphaChip
#79Earlier quoted context omitted.
There are benchmarks in this space. You can also bring your chip designs into the open and show what happens with different tools. You can run the algorithm on the placed designs that you sponsor for open source VLSI to show how much better they are. None of this has been done. This is table stakes if you want to talk about your EDA algorithm advancement. If this weren't coming out of Google, everybody would laugh it…
> Nothing about AlphaChip even reaches ordinary evidence. You reply is wildly confident and dismissive. If correct, why did Nature choose to publish?