Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jeff Dean responds to EDA industry about AlphaChip

twitter.com

Post reply on HN
Print view View original

161–170 of 221 posts

Re: Jeff Dean responds to EDA industry about AlphaChip

#161

Post by wholehog »

seanhunter wrote: post #110
xpe wrote: post #17

Earlier quoted context omitted.

> Google continues to peddle unsubstantiated snake oil I read your comment, but I'm not following -- or maybe I disagree with it -- I'm not sure yet. "Snake oil" is an emotionally loaded term that raises the temperature of the conversation. That usually makes having a conversation harder. From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. Agree? Are you claiming that AlphaChip is not?…

Well here’s one exaggeration that was pretty obvious to me straight away as a somewhat disinterested observer. In her status on X Anna Goldie says [1] “ AlphaChip was one of the first RL methods deployed to solve a real-world engineering problem”. This seems very clearly untrue- for example here’s a real-world engineering use of reinforcement learning by google AI themselves from 6 years ago [2] which if you use Anna…

That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.

Re: Jeff Dean responds to EDA industry about AlphaChip

#162

Post by wholehog »

dogleg77 wrote: post #148
btilly wrote: post #72

Earlier quoted context omitted.

Look, either the follow-up article did pretraining or not. Jeff Dean is claiming that the importance of pretraining was mentioned 37 times and the follow-up didn't do it. That sounds easy to verify. Likewise the importance of spending 20x as much money on the training portion seems easy to verify, and significant. That they would fail to properly test against industry standard workbenches seems reasonable to me. This…

Reading those papers and looking at the code, it doesn't look easy. However, let's imagine that the Cheng et al team comes back with results for pretraining a few months from now, and they support the conclusions of their earlier paper. What should they do to help everyone reach a conclusion?

Pre-training is just training on multiple chips.

"If Cheng et al. had reached out to the corresponding authors of the Nature paper, we would have gladly helped them to correct these issues prior to publication" (https://arxiv.org/pdf/2411.10053)

That's how you actually do a reproduction study - you reach out to the corresponding authors and make sure you do everything exactly the same. But at this point, it's hard to imagine the AlphaChip folks having much patience with them.

Re: Jeff Dean responds to EDA industry about AlphaChip

#163

Post by wholehog »

bsder wrote: post #62
btilly wrote: post #49

Earlier quoted context omitted.

And do you believe that that is what's happening in this case? If you have personal experience with Jeff Dean et al that you're willing to share, I'd be interested in hearing about it. From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss." It's not even clea…

> From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss." From where I'm sitting it looks like Google cooked the books maximally, barely beat humans let alone state of the art algorithms, published a crappy article in Nature because it would never have passed…

> published a crappy article in Nature because it would never have passed editorial muster at something like DAC or an IEEE journal and now have to browbeat other people who are calling them out on it.

I don't think it's easier to get into DAC / an IEEE journal than Nature.

Their human baseline was the TPU physical design team, with access to the best available tools: rdcu.be/cmedX

and this is still the baseline to beat in order to get used in production, which has happened for multiple generations of TPU.

TPU is export controlled and super confidential -- multi-billion dollar IP! -- so I don't see raw data coming out anytime soon.

Re: Jeff Dean responds to EDA industry about AlphaChip

#164

Post by pclmulqdq »

wholehog wrote: post #155
pclmulqdq wrote: post #103

Earlier quoted context omitted.

Yes, they are. The other approaches usually look like simulated annealing, which has several hyperparameters that control how much computing is used and improve results with more compute usage.

See my comment above - the Nature authors already did this, and tried a huge hyperparameter sweep for SA, and RL still won. See appendix of the Nature article: rdcu.be/cmedX

I understand and have read the article. Running 80 experiments with a crude form of simulated annealing is at most 0.0000000001% of the effort that has been spent on making that kind of hill climb work well by traditional EDA vendors. That is also an in-sample comparison, where I would believe the Google thing pre-trained on Google chips would do well, while it might have a harder time with a chip designed by a third party (further from its pre-training).

The modern versions of that hill climb also use some RL (placing and routing chips is sort of like a game), but not in the way Jeff Dean wants it to be done.

Re: Jeff Dean responds to EDA industry about AlphaChip

#165

Post by wholehog »

phonon wrote: post #99
jeffbee wrote: post #80

Earlier quoted context omitted.

The 2022 paper pretty explicitly says that runtime is not a substitute. They say their best result "can only be achieved in our 8-GPU setup".

I assume you mean Fig. 6 here?[0] But that was explicitly limited to 8 hours for all setups. Do they have another paper that shows that you can't increase the number of hours of a smaller GPU setup to compensate? [0] https://dl.acm.org/doi/pdf/10.1145/3505170.3511478

They also changed the ratio of RL experience collectors to GPU workers (~1/20th the RL experience collectors, 1/2 the GPUs). I don't know what impact that has --- maybe each GPU episode has less experience? Maybe that makes for an effectively small batch size and therefore more chaotic training? But either way, why change things when you can just match them exactly?

Re: Jeff Dean responds to EDA industry about AlphaChip

#166

Post by sijnapq »

vighneshiyer wrote: post #122
Jabbles wrote: post #98

Earlier quoted context omitted.

You're saying that if the other methods were given the equivalent amount of compute they might be able to perform as well as AlphaChip? Or at least that the comparison would be fairer? Are the other methods scalable in that way?

Existing mixed-placement algorithms depend on hyperparameters, heuristics, and initial states / randomness. If afforded more compute resources, they can explore a much wider space and in theory come up with better solutions. Some algorithms like simulated annealing are easy to modify to exploit arbitrarily more compute resources. Indeed, I believe the comparison of AlphaChip to alternatives would be fairer if compute…

[dead]

Re: Jeff Dean responds to EDA industry about AlphaChip

#167

Post by sijnapq »

wholehog wrote: post #157
dogleg77 wrote: post #143

Earlier quoted context omitted.

That's an appeal to authority, and not an effective one. Jeff Dean doesn't have a good track record in chip design.

What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.

[dead]

Re: Jeff Dean responds to EDA industry about AlphaChip

#168

Post by menaerus »

solidasparagus wrote: post #118
menaerus wrote: post #96

Earlier quoted context omitted.

Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day income?

Who cares? That's someone else's problem. I just pay 2-3$/hr and the H100s are usable

Look, I understand that some people are short-sighted and can hardly think out of the box and that is totally fine by me. I don't judge you for being that so I kindly ask you not to judge my question. Learn to give some benefit of the doubt.

Re: Jeff Dean responds to EDA industry about AlphaChip

#169

Post by amelius »

woodson wrote: post #141
amelius wrote: post #127

Earlier quoted context omitted.

Yes, the community should force Nature to up its standards or ditch it. Software replication should be trivial in this day and age.

All these papers doing "research" on how to better prompt ChatGPT would be unpublishable then, given that API access to older models gets retired, so the findings of these papers can no longer be reproduced. (I agree with you in principle; my example above is meant to show that standards for things such as reproducibility aren't easily defined. There are so many factors to consider.)

Well since you put "research" in quotes, I think you also agree that this type of work does not really belong in a quality journal with a high impact factor ;)

Re: Jeff Dean responds to EDA industry about AlphaChip

#170

Post by seanhunter »

wholehog wrote: post #161
seanhunter wrote: post #110

Earlier quoted context omitted.

Well here’s one exaggeration that was pretty obvious to me straight away as a somewhat disinterested observer. In her status on X Anna Goldie says [1] “ AlphaChip was one of the first RL methods deployed to solve a real-world engineering problem”. This seems very clearly untrue- for example here’s a real-world engineering use of reinforcement learning by google AI themselves from 6 years ago [2] which if you use Anna…

That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.

Agreed but if someone at your own company did it two years before you in the context of something that recent it’s stretching credibility to say you were one of the first.
Post reply on HN