Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jeff Dean responds to EDA industry about AlphaChip

twitter.com

Post reply on HN
Print view View original

61–70 of 221 posts

Re: Jeff Dean responds to EDA industry about AlphaChip

#61

Post by throwaway2037 »

quadrature wrote: post #45
AtlasBarfed wrote: post #41

How the hell would you verify an AI-generated silicon design? Like, for a CPU, you want to be sure it behaves properly for the given inputs. Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs? I mean, I guess if the chip is designed for AI, and AIs are inherently nonguaranteed output/responses, then the AI chip design being nonguaranteed isn't any difference in nonguarantees. Unless it i…

> How the hell would you verify an AI-generated silicon design? I think you're asking a different question, but in the context of the OP researchers are exploring AI for solving deterministic but intractable problems in the field of chip design and not generating designs end to end. Here's an excerpt from the paper. "The objective is to place a netlist graph of macros (e.g., SRAMs) and standard cells (logic gates, su…

    > Despite decades of research on this problem, it is still necessary for human experts to iterate for weeks with the existing placement tools, in order to produce solutions that meet multi-faceted design criteria.
Ironically, this sounds a lot like building a bot to play StarCraft, which is exactly what AlphaStar did. I had no idea that EDA layout is still so difficult and manual in 2024. This seems like a very worth area of research.

I am not an expert in AI/ML, but is the ultimate goal: Train on as many open source circuit designs as possible to build a base, then try to solve IC layouts problems via reinforcement learning, similar to AlphaStar. Finally, use the trained model to do inference during IC layout?

Re: Jeff Dean responds to EDA industry about AlphaChip

#62

Post by bsder »

btilly wrote: post #49
bsder wrote: post #40

Earlier quoted context omitted.

> Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other? Because lots of engineers are being told by managers "Why aren't we using that tool?" and a bunch of engineers are stuck saying "Because it doesn't actually work." aka "Google is lying through their teeth." to which the response is "Oh, so you know better than Google?" to which the reponse is "Yeah, actual…

And do you believe that that is what's happening in this case? If you have personal experience with Jeff Dean et al that you're willing to share, I'd be interested in hearing about it. From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss." It's not even clea…

> From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss."

From where I'm sitting it looks like Google cooked the books maximally, barely beat humans let alone state of the art algorithms, published a crappy article in Nature because it would never have passed editorial muster at something like DAC or an IEEE journal and now have to browbeat other people who are calling them out on it.

And that's the best interpretation we can cough up.

I'll go further, we don't even have any raw data that says that they actually did beat the humans. Some of the humans I know who run P&R are REALLY good at what they do. The data could be completely made up. Given how much scientific fraud has come out lately, I'm amazed at the number of people defending Google on this.

Where I'm from, we call what Google is doing both "lying" and "bullying".

Look, Google can easily defuse this in all manner of ways. Publish their raw data. Run things on testbenches and benchmarks that the EDA tools vendors have been running on for years. Run things on the open source VLSI designs that they sponsored.

What I suspect happened is that Google's AI group has gotten used to being able to make hyperbolic marketing claims which are difficult to verify. They then poked at place and route, failed, and published an article anyway because someone's promotion is tied to this. They expected that everybody would swallow their glop just like every other time, be mostly ignored and the people involved can get their promotions and move on.

Unfortunately, Google is shoveling bullshit around something that has objective answers; real money is at stake; and they're getting rightfully excoriated for it.

Whoops.

Re: Jeff Dean responds to EDA industry about AlphaChip

#63

Post by throwaway2037 »

bsder wrote: post #4

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…

    > EDA companies are garbage
I don't understand this comment. Can you please explain? Are they unethical? Or do they write poor software?

Re: Jeff Dean responds to EDA industry about AlphaChip

#64

Post by throwaway2037 »

stackghost wrote: post #48
rajup wrote: post #32

Earlier quoted context omitted.

Really not sure how you’re conflating product demos which are known to be pie in the sky across the industry (not just Google) with peer reviewed research published in journals. Super basic distinction imho.

>peer reviewed research published in journals Peer review doesn't mean as much as Elsevier would like you to believe. Plenty of peer-reviewed research is absolute trash.

All of the highest impact papers authored by DeepMind and Google Brain have appeared in Nature, which is the gold standard for peer-reviewed natural science research. What exactly are you trying to claim about Google's peer-reviewed papers?

Re: Jeff Dean responds to EDA industry about AlphaChip

#65

Post by YetAnotherNick »

bushbaba wrote: post #53
wholehog wrote: post #50

Earlier quoted context omitted.

We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the…

h100 GPU instances are multiple orders of magnitude more expensive.

H100 GPUs are more or less similar in price/performance. It is 2-3x more expensive per hour for 2-3x higher performance.

Re: Jeff Dean responds to EDA industry about AlphaChip

#66

Post by throwaway2037 »

bsder wrote: post #36
joshuamorton wrote: post #7

Earlier quoted context omitted.

As someone who has no skin in the game and is only loosely following this, there is a tool: https://github.com/google-research/circuit_training , the detractors claim to not be able to reproduce Google's results (what Dean is commenting on) with it, Google and 1-2 other companies claim to be using it internally to success (e.g. see the end of this article: https://deepmind.google/discover/blog/how-alphachip-transfor.…

There are benchmarks in this space. You can also bring your chip designs into the open and show what happens with different tools. You can run the algorithm on the placed designs that you sponsor for open source VLSI to show how much better they are. None of this has been done. This is table stakes if you want to talk about your EDA algorithm advancement. If this weren't coming out of Google, everybody would laugh it…

    > Nothing about AlphaChip even reaches ordinary evidence.
You reply is wildly confident and dismissive. If correct, why did Nature choose to publish?

Re: Jeff Dean responds to EDA industry about AlphaChip

#67

Post by vighneshiyer »

wholehog wrote: post #50
vighneshiyer wrote: post #38

I have published an addendum to an article I wrote about AlphaChip ( https://vighneshiyer.com/misc/ml-for-placement/ ) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still ma…

We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the…

You are correct. For commercial use, the GPUs used for training and fine-tuning aren't a problem financially. However, if we wanted to rigorously benchmark AlphaChip against simulated annealing or other floorplanning algorithms, we have to afford the same compute and runtime budget to each algorithm. With 16 GPUs running for 6 hours, you could explore a huge placement space using any algorithm, and it isn't clear if RL will outperform the other ones. Furthermore, the runtime of AlphaChip as shown in the Nature paper and ISPD was still significantly greater than Cadence's concurrent macro placer (even after pre-training, RL requires several hours of fine-tuning on the target problem instance). Arguably, the runtime could go down with more GPUs, but at this point, it is unclear how much value is coming from the policy network / problem embedding vs the ability to explore many potential placements.

Re: Jeff Dean responds to EDA industry about AlphaChip

#68

Post by bsder »

throwaway2037 wrote: post #63
bsder wrote: post #4

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…

> EDA companies are garbage I don't understand this comment. Can you please explain? Are they unethical? Or do they write poor software?

Yes and yes.

EDA companies are gatekeeping monopolies. They absolutely abuse their monopoly position to extract huge chunks of money out of companies, and are pretty much single-handedly responsible for the fact that the hardware startup ecosystem is moribund compared to that of the software startup ecosystem.

They have been horrible liars about performance and benchmarketing for decades. They dragged their feet miserably over releasing Linux versions of their software because they were extracting money based upon number of CPU licenses (everything was on Sparc which was vastly inferior). Their software hasn't really improved all that much over decades--mostly they benefited from Moore's Law. They have made a point of stifling attempts at interoperability and open data exchange. They have bought lots of competitors mostly to just shut them down. I can go on and on.

The EDA companies aren't quite Oracle--but they're not far off.

This is one of the reasons why Google is getting pounded over this--maybe even unfairly. People in the field are super sensitive about bullshit claims from EDA vendors--we've heard them all and been on the receiving end of the stick far too many times.

Re: Jeff Dean responds to EDA industry about AlphaChip

#69

Post by throwaway2037 »

lobochrome wrote: post #56
bsder wrote: post #4

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is…

Agreed, in particular on #2 Given infinite time and compute - maybe the approach is significantly better. But that’s just not practical. So unless you see dramatic shifts - no one is going to throw away proven results on your new approach because of the TTM penalty if it goes wrong. The EDA industry is (has to be) ultra conservative.

    > The EDA industry is (has to be) ultra conservative.
What is special about EDA that requires it to be more conservative?

Re: Jeff Dean responds to EDA industry about AlphaChip

#70

Post by nemonemo »

vighneshiyer wrote: post #38

I have published an addendum to an article I wrote about AlphaChip ( https://vighneshiyer.com/misc/ml-for-placement/ ) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still ma…

In the conclusion of the article, you said: "While I concede that there are things the ISPD authors could have done better, their conclusion is still sound. The Nature authors do not address the fact that CMP and AutoDMP outperform CT with far less runtime and compute requirements."

One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this?

Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article toward the AlphaChip work (words like "arrogance", "disdain", "hyperbole", "belittling", "hostile" for AlphaChip authors, as opposed to "excellent" for a Synopsys VP.) Could you share where this difference in tone originates?

Post reply on HN