Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jeff Dean responds to EDA industry about AlphaChip

twitter.com

Post reply on HN
Print view View original

201–210 of 221 posts

Re: Jeff Dean responds to EDA industry about AlphaChip

#201

Post by dogleg77 »

nemonemo wrote: post #187
dogleg77 wrote: post #176

Earlier quoted context omitted.

I am trying to understand what you mean here by potential to disrupt. AlphaChip addresses one out of hundreds of tasks in chip design. Macro placement is a part of mixed-size placement, which is handled just fine by existing tools, many academic tools, open-source tools, and Nvidia AutoDMP. Even if AlphaChip was commonly accepted as a breakthrough, there is no disruption here. Direct comparisons from the last 3 years…

> Direct comparisons from the last 3 years show that AlphaChip is worse. Do you have any evidence to claim this? The whole point of this thread is that the direct comparisons might have been insufficient, and even the author of "The Saga" article who's biased against the AlphaChip work agreed. > Granted, Google is belittling these comparisons, but that's what you'd expect. This kind of language doesn't help any posit…

I am referring to direct comparisons in Cheng et al and in Stronger Baselines that everyone is discussing. Let's assume your point about "might have been insufficient". We don't currently have the luxury to be frequentists, as we don't have many academic groups reporting results for running Google code. From the Bayesian perspective, that's the evidence we have.

Maybe you know more such published papers than I do, or you know the reasons why there aren't many. Somehow this lack of follow-up over three years suggests a dead-end.

As for "belittle", how would you describe the scientific term "regurgitating" used by Jeff Dean? Also, the term "fundamentally flawed" in reference to a 2023 paper by two senior professors with serious expertise and track record in the field, that for some reason no other experts in the field criticize? Where was Jeff Dean when that paper was published and reported by the media?

Unless Cheng and Kahng agree with this characterization, Jeff Dean's timing and language are counterproductive. If he ends up being wrong on this, what's the right thing to do?

Re: Jeff Dean responds to EDA industry about AlphaChip

#202

Post by nemonemo »

nemonemo wrote: post #187
dogleg77 wrote: post #176

Earlier quoted context omitted.

I am trying to understand what you mean here by potential to disrupt. AlphaChip addresses one out of hundreds of tasks in chip design. Macro placement is a part of mixed-size placement, which is handled just fine by existing tools, many academic tools, open-source tools, and Nvidia AutoDMP. Even if AlphaChip was commonly accepted as a breakthrough, there is no disruption here. Direct comparisons from the last 3 years…

> Direct comparisons from the last 3 years show that AlphaChip is worse. Do you have any evidence to claim this? The whole point of this thread is that the direct comparisons might have been insufficient, and even the author of "The Saga" article who's biased against the AlphaChip work agreed. > Granted, Google is belittling these comparisons, but that's what you'd expect. This kind of language doesn't help any posit…

> direct comparisons in Cheng

That's the ISPD paper referenced many times in this whole thread.

> Stronger Baselines

Re: "Stronger baselines", the paper "That Chip Has Sailed" says "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline." What is your take on this claim?

As for 'regurgitating,' I don’t think it helps Jeff Dean’s point either. Based on my and vighneshiyer's discussion above, describing the work as "fundamentally flawed" does not seem far-fetched. If Cheng and Kahng do not agree with this, I believe they can publish another invited paper.

On 'belittle,' my main issue was with your follow-up phrase, 'that’s what you’d expect.' It comes across as overly emotional and detracts from the discussion.

Regarding lack of follow-ups (I am aware of), the substantial resources required for this work seem beyond what academia can easily replicate. Additionally, according to "the Saga" article, both non-Jeff Dean authors have left Google until recently, but their Twitter/X/LinkedIn seem to say they came back to Google and seem to have worked on this "Sailing Chip" paper.

Personally, I hope they reignite their efforts on RL in EDA and work toward democratizing their methods so that other researchers can build new systems on their foundation. What are your thoughts? Do you hope they improve and refine their approach in future work, or do you believe there should be no continuation of this line of research?

Re: Jeff Dean responds to EDA industry about AlphaChip

#203

Post by seanhunter »

wholehog wrote: post #190
seanhunter wrote: post #170

Earlier quoted context omitted.

Agreed but if someone at your own company did it two years before you in the context of something that recent it’s stretching credibility to say you were one of the first.

I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.

Yes. The sky is also blue?

However it's hard to see how being provably 2 years behind the first even in your own company in an incredibly hot area that people are doing tons of work in makes you suddenly second. By that logic I might still be in time to claim the silver for the 100m at the Paris olympics if I pop over there in the next 18 months or so.

I can see you created this account just to comment on this thread so I'm sure you have more inside information than I do given that I'm really not connected to this in any way. Enjoy your work at Google Research. I think you guys do cool stuff. It's a shame in my opinion that you choose to damage your credibility by making (and defending) such obviously false claims rather than concentrating on the genuinely innovative work you have done advancing the field.

Re: Jeff Dean responds to EDA industry about AlphaChip

#204

Post by LittleTimothy »

"You didn't use enough compute in your reproduction of our methods" is kind of a funny criticism today. Well yeah, you're Google. Sorry guys, no reviewing of our methodology unless you own a cloud. It wouldn't surprise me if it's true that more compute, more pre-training etc. provides a lot of utility, but that does make it difficult to verify the work.

One interesting aspect of this though is vice-versa, whilst Google has oodles of compute, Synopsys has oodles of data to train on (if, and this is a massive if, they can get away with training on customer IP).

Re: Jeff Dean responds to EDA industry about AlphaChip

#205

Post by LittleTimothy »

wholehog wrote: post #160
rowanG077 wrote: post #78

I don't get. Why isn't the model open if it works? If it isn't this is just a fart in the wind. If it is the findings should be straightforward to replicate.

It is open: https://github.com/google-research/circuit_training

As far as I understand it, only kind of? It's open source, but in their paper they did a tonne of pre-training and whilst they've released a small pre-training checkpoint they haven't released the results of the pre-training they've done for their paper. So anyone reproducing this will innevitably be accused of failing to pretrain the model correctly?

Re: Jeff Dean responds to EDA industry about AlphaChip

#206

Post by dogleg77 »

wholehog wrote: post #199
dogleg77 wrote: post #193

Earlier quoted context omitted.

You must be kidding. They did reach out and documented their interactions, with names of engineers, dates, and all. Someone is ghosting someone here.

"These major methodological differences unfortunately invalidate Cheng et al.’s comparisons with and conclusions about our method. If Cheng et al. had reached out to the corresponding authors of the Nature paper[8], we would have gladly helped them to correct these issues prior to publication[9]. [8] Prior to publication of Cheng et al., our last correspondence with any of its authors was in August of 2022 when we re…

That is misleading. The first two authors left Google in August 2022 under unclear circumstances. The code and data were owned by Google, that's probably why Kahng continued discussibg code and data with his Google contacts. He received clear answers from several Google employees, so if they were at fault, Google should apologize rather than blame Cheng and Kahng.

Re: Jeff Dean responds to EDA industry about AlphaChip

#207

Post by dogleg77 »

wholehog wrote: post #181
dogleg77 wrote: post #178

Earlier quoted context omitted.

Something is off with your source references. Chatterjee's amended complaint is a legal document filed under penalty of perjury, accepted by a US judge, available publicly, apparently not "thrown out". How does an earlier document figure into this? How do we know it was "thrown out" and for what reason? Obviously, a later document is what matters. Also, you are using an unreviewed document from Google not published i…

His original complaint being dismissed matters because it suggests that he was fishing around for a complaint that was valid, and that perhaps his primary motivation was to get money out of Google. Legal nitpick - you can get away with alleging pretty much whatever you want in a legal complaint. You can't even be sued for defamation if it turns out later you were lying. Jeff Dean isn't saying that Cheng et al. should…

No, it doesn't suggest that. Complaints are often dismissed on technicalities or because they are written poorly.

Google claimed their new algorithm as a breakthrough. If this were the so, the algorithm would have helped design chips in many different cases. Now, the defense is that it only works for some inputs, and those inputs cannot be shared. This is not a serious defense and looks like a coverup.

Re: Jeff Dean responds to EDA industry about AlphaChip

#208

Post by dogleg77 »

LittleTimothy wrote: post #204

"You didn't use enough compute in your reproduction of our methods" is kind of a funny criticism today. Well yeah, you're Google. Sorry guys, no reviewing of our methodology unless you own a cloud. It wouldn't surprise me if it's true that more compute, more pre-training etc. provides a lot of utility, but that does make it difficult to verify the work. One interesting aspect of this though is vice-versa, whilst Goog…

Chip designers run EDAtools on premises. How would EDA companies have access to customer data? Maybe for debugging purposes under NDA, but that won't allow training on customer IP.

Re: Jeff Dean responds to EDA industry about AlphaChip

#209

Post by cosentiyes »

otterley wrote: post #174
wholehog wrote: post #157

Earlier quoted context omitted.

What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.

By what measure are TPUs “successful”? Where is your data coming from?

They're the only non-nvidia accelerator used to train state of the art large language models at scale?

Re: Jeff Dean responds to EDA industry about AlphaChip

#210

Post by dogleg77 »

nemonemo wrote: post #202
nemonemo wrote: post #187

Earlier quoted context omitted.

> Direct comparisons from the last 3 years show that AlphaChip is worse. Do you have any evidence to claim this? The whole point of this thread is that the direct comparisons might have been insufficient, and even the author of "The Saga" article who's biased against the AlphaChip work agreed. > Granted, Google is belittling these comparisons, but that's what you'd expect. This kind of language doesn't help any posit…

> direct comparisons in Cheng That's the ISPD paper referenced many times in this whole thread. > Stronger Baselines Re: "Stronger baselines", the paper "That Chip Has Sailed" says "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline." What is your take on this claim? As for 're…

The point is that the Cheng et al results and paper were shown to Google and apparently okayed by Google points of contact. After this, complaining that Cheng et al didn't ask someone outside Google makes little sense. These far fetched excuses and emotional wording by Jeff Dean leave a big cloud over the Nature work. If he is confident everything is fine, he would not bother.

To clarify "you'd expect" - if Jeff Dean is correct, he'd deny problems and if he's wrong he'd deny problems. So, his response carries little information. Rationally, this should be done by someone else with a track record in chip implementation.

Post reply on HN