Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jeff Dean responds to EDA industry about AlphaChip

twitter.com

Post reply on HN
Print view View original

181–190 of 221 posts

Re: Jeff Dean responds to EDA industry about AlphaChip

#181

Post by wholehog »

dogleg77 wrote: post #178
wholehog wrote: post #158

Earlier quoted context omitted.

You're linking to his amended complaint - his original complaint was thrown out because it alleged things like "Google's motto is don't be evil, but they were evil, thus defrauding me." According to a Google investigator's sworn statement, he admitted that he didn't have evidence to suspect the AlphaChip authors of fraud: "he stated that he suspected that the research being conducted by Goldie and Mirhoseini was frau…

Something is off with your source references. Chatterjee's amended complaint is a legal document filed under penalty of perjury, accepted by a US judge, available publicly, apparently not "thrown out". How does an earlier document figure into this? How do we know it was "thrown out" and for what reason? Obviously, a later document is what matters. Also, you are using an unreviewed document from Google not published i…

His original complaint being dismissed matters because it suggests that he was fishing around for a complaint that was valid, and that perhaps his primary motivation was to get money out of Google.

Legal nitpick - you can get away with alleging pretty much whatever you want in a legal complaint. You can't even be sued for defamation if it turns out later you were lying.

Jeff Dean isn't saying that Cheng et al. should be unpublished; he's saying that they didn't run the method the same way. It is perfectly fine for someone to try changing the method and report what they found. What's not fine is to claim that this means that Google was lying in their study.

Re: Jeff Dean responds to EDA industry about AlphaChip

#182

Post by 11101010001100 »

xpe wrote: post #173
11101010001100 wrote: post #120

Earlier quoted context omitted.

The paper is more or less a dead end. If there is another name you want to call it, by all means.

/[01]{8,}/: I was hoping to have a conversation. This is why I asked questions. Any responses to them? Looking up the thread, you can see the context. Many of us pushed back against vague claims that AlphaChip was "snake oil". Like good engineers, we split apart the problem into clearer concepts. The "snake oil" proponents did not offer compelling replies, did they? Instead, they retreated to irrelevant points that h…

MRS is this week, you can go and join the conversations with people at the metal level. Probably even talk to the authors themselves!

Re: Jeff Dean responds to EDA industry about AlphaChip

#183

Post by wholehog »

dogleg77 wrote: post #175
wholehog wrote: post #156

Earlier quoted context omitted.

> Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. Was the TPU physical design team also taken in? And also MediaTek? And also TF-Agents, which publicly said they re-produced the AlphaChip method and results exactly?

What did the TPU physical design team say about this publicly? Can you also point to a statement from MediaTek? (I've seen a quote in Google blog, but was unable to confirm it). Who in the TF-agents team has serious physical design background?

Are you really suggesting that the TPU team does not stand behind the graphs in Google's own blog post? And that MediaTek does not stand behind their quoted statement?

Re: Jeff Dean responds to EDA industry about AlphaChip

#184

Post by wholehog »

boredatoms wrote: post #74
_cs2017_ wrote: post #59

Curious why there's so much emotion and unpleasantness in this dispute? How did it evolve from the boring academic argument about benchmarks, significance, etc to a battle of personal attacks?

A lot of people work on non-AI implementations

Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

Re: Jeff Dean responds to EDA industry about AlphaChip

#185

Post by wholehog »

oesa wrote: post #86

In the tweet Jeff Dean says that Cheng at al. failed to follow the steps required to replicate the work of the Google researchers. Specifically: > In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), robbing our learning-based method of its ability to learn from other chip designs But in the Circuit Training Google repo[1] they specifically say: > Our re…

> Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction?

The Circuit Training repo was just going through an example. It is common for an open-source repo to describe simple examples for testing / validating your setup --- that does not mean this is how you should get optimal results in general. The confusion may stem from their statement that, in this example, they produced results that were comparable with the pre-trained results in the paper. This is clearly not a general repudiation of pre-training.

If Cheng et al. genuinely felt this was ambiguous, they should have reached out to the corresponding authors. If they ran into some part of the repo they felt they had to "reverse-engineer", they should have asked about that, too.

Re: Jeff Dean responds to EDA industry about AlphaChip

#186

Post by wholehog »

nemonemo wrote: post #136
vighneshiyer wrote: post #121

Earlier quoted context omitted.

Cadence in particular has been quite receptive to allowing academics and researchers to benchmark new algorithms against their tools. They have also been quite permissive with letting people publish TCL scripts for their tools ( https://github.com/TILOS-AI-Institute/MacroPlacement/tree/ma... ) that in theory should enable precise reproduction of results. From my knowledge, Cadence has been very permissive from 2022 o…

We're not just talking about academia—Google's AlphaChip has the potential to disrupt the balance of the EDA industry's duopoly. It seems unlikely that Google could easily secure the policy or license changes necessary to publish direct comparisons in this context. If publicizing comparisons of CMPs is as permissible as you suggest, have you seen a publication that directly compares a Cadence macro placement tool wit…

> Does the EDA ecosystem support a similarly open culture of benchmarking for commercial tools?

If only. The comparison in Cheng et al. is the only public comparison with CMP that I can recall, and it is pretty suss that this just so happens to be a very pro-commercial-autoplacer study. (And, Cheng et al. have cited 'licensing agreements' as a reason for not giving out the synthesized netlists necessary to reproduce their results.)

Reminded a bit of Oracle. They likewise used to (and maybe still?) prohibit any benchmarking of their database software against that of another provider. This seems to be a common move for solidifying a strong market position.

Re: Jeff Dean responds to EDA industry about AlphaChip

#187

Post by nemonemo »

dogleg77 wrote: post #176
nemonemo wrote: post #136

Earlier quoted context omitted.

We're not just talking about academia—Google's AlphaChip has the potential to disrupt the balance of the EDA industry's duopoly. It seems unlikely that Google could easily secure the policy or license changes necessary to publish direct comparisons in this context. If publicizing comparisons of CMPs is as permissible as you suggest, have you seen a publication that directly compares a Cadence macro placement tool wit…

I am trying to understand what you mean here by potential to disrupt. AlphaChip addresses one out of hundreds of tasks in chip design. Macro placement is a part of mixed-size placement, which is handled just fine by existing tools, many academic tools, open-source tools, and Nvidia AutoDMP. Even if AlphaChip was commonly accepted as a breakthrough, there is no disruption here. Direct comparisons from the last 3 years…

> Direct comparisons from the last 3 years show that AlphaChip is worse.

Do you have any evidence to claim this? The whole point of this thread is that the direct comparisons might have been insufficient, and even the author of "The Saga" article who's biased against the AlphaChip work agreed.

> Granted, Google is belittling these comparisons, but that's what you'd expect.

This kind of language doesn't help any position you want to advocate.

About "the potential to disrupt", a potential is a potential. It's an initial work. What I find interesting is that people are so eager to assert that it's a dead-end without sufficient exploration.

Re: Jeff Dean responds to EDA industry about AlphaChip

#188

Post by wholehog »

segmondy wrote: post #117

It's ridiculous how expensive the wrong hire can be https://www.wired.com/story/google-brain-ai-researcher-fired...

So much wasted time.

He even ran a study internally (with Markov), but, as the AlphaChip authors describe:

In 2022, it was reviewed by an independent committee at Google, which determined that “the claims and conclusions in the draft are not scientifically backed by the experiments” [33] and “as the [AlphaChip] results on their original datasets were independently reproduced, this brought the [Markov et al.] RL results into question” [33]. We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper. (https://arxiv.org/pdf/2411.10053)

Re: Jeff Dean responds to EDA industry about AlphaChip

#189

Post by pclmulqdq »

wholehog wrote: post #180
pclmulqdq wrote: post #164

Earlier quoted context omitted.

I understand and have read the article. Running 80 experiments with a crude form of simulated annealing is at most 0.0000000001% of the effort that has been spent on making that kind of hill climb work well by traditional EDA vendors. That is also an in-sample comparison, where I would believe the Google thing pre-trained on Google chips would do well, while it might have a harder time with a chip designed by a third…

> it might have a harder time with a chip designed by a third party (further from its pre-training). Then they could pre-train on chips that are in-distribution for that task. See also section 3.1 of their response paper, where they describe a comparison against commercial autoplacers: https://arxiv.org/pdf/2411.10053

The comparison in that paper was very much not fair to Google's method. Google's original published comparison to simulated annealing is not fair to simulated annealing methods. That is, unfortunately, part of the game of publication when you want to publish a marginal result.

It is possible that the pre-training step may overfit to a particular class of chips or may fail to converge given a general sample of chip designs. That would make the pre-training step unable to be used in the setting of a commercial EDA tool. The people who do know this are the people at EDA companies who are smart and not arrogant and who benchmarked this stuff before deciding not to adopt it.

If you want to make a good-faith assumption (that IMO is unwarranted given the rest of the paper), the people trying to replicate Google's paper may have done a pre-training step that failed to converge, and then didn't report it. That failure to converge could be due to ineptitude, but it could be due to data quality, too.

Re: Jeff Dean responds to EDA industry about AlphaChip

#190

Post by wholehog »

seanhunter wrote: post #170
wholehog wrote: post #161

Earlier quoted context omitted.

That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.

Agreed but if someone at your own company did it two years before you in the context of something that recent it’s stretching credibility to say you were one of the first.

I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.
Post reply on HN