Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Trains LLM on Chip Design

eetimes.com

Post reply on HN
Print view View original

31–40 of 151 posts

Re: Nvidia Trains LLM on Chip Design

#31

Post by anon291 »

dddrh wrote: post #9

Interesting concept that raised the question for me: What is the primary limiting factor right now that prevents LLM’s or any other AI model to go “end to end” on programming a full software solution or full design/engineering solution? Is it token limitations or accuracy the further you get into the solution?

The issue with transformers is the context length. Compute wise, we can figure out the long context window (in terms of figuring out the attention matrix and doing the calculations). The issue is training. The weights are specialized to deal with contexts only of a certain size. As far as I know, there's no surefire solution that can overcome this. But theoretically, if you were okay with the quadratic explosion (and had a good dataset, another point...) you could spend money and train it for much longer context lengths. I think for a full project you'd need millions of tokens.

Re: Nvidia Trains LLM on Chip Design

#32

Post by anon291 »

margorczynski wrote: post #17
drsopp wrote: post #13

Earlier quoted context omitted.

I guess this would be the context window size in the case of LLMs. Edit: On second thought, maybe at a certain minimum context window size it is possible to cajole the instructions in such a way that you at any point in the process make the LLM work at a suitable level of abstraction more like humans do.

Maybe the issue is that for us the "context window" that we feed ourselves is actually a compressed and abstracted version - we do not re-feed ourselves the whole conversation but a "notion" and key points that we have stored. LLMs have static memory so I guess there is no other way as to single-pass the whole thing. For human-like learning it would need to update it state (learn) on the fly as it does inference.

See https://github.com/mit-han-lab/streaming-llm and others. There's good reason to believe that attention networks learn how to update their own weights (Forget the paper) based on their input. The attention mechanism can act like a delta to update weights as the data propagates through the layers. The issue is getting the token embeddings to be more than just the 50k or so that we use for the english language so you can explore the full space, which is what the attention sink mechanism is trying to do.

Re: Nvidia Trains LLM on Chip Design

#33

Post by passion__desire »

xnx wrote: post #7

Google has been using machine learning for chip design since at least 2021: https://www.nature.com/articles/s41586-021-03544-w Hasn't brought about the singularity yet.

Google’s Chip Designing AI - https://www.youtube.com/watch?v=zR9IusOpEzk

Analog Chip Design is an Art. Can AI Help? - https://www.youtube.com/watch?v=lNypq1XuZRo

Re: Nvidia Trains LLM on Chip Design

#34

Post by HPsquared »

gumby wrote: post #6
hasbot wrote: post #4

The title is a bit misleading as the first sentence says "to help chip designers with tasks related to chip design, including answering general questions about chip design, summarizing bug documentation, and writing scripts for EDA tools." Still pretty cool though.

Isn’t that what chip design is?

It's a bit like "assistant manager" vs "assistant to the manager".

Re: Nvidia Trains LLM on Chip Design

#35

Post by ethanbond »

falcor84 wrote: post #22

"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind... Thus the first ultraintellig…

Don’t worry, it’ll be good because it’s trained on human stories, in which usually the good guy wins.

Re: Nvidia Trains LLM on Chip Design

#36

Post by ethanbond »

hasbot wrote: post #8
gumby wrote: post #6

Earlier quoted context omitted.

Isn’t that what chip design is?

The title suggested to me, and I see other commenters here, that the LLM was doing the chip design which isn't the case at all. So misleading title.

I remember seeing a tweet from an AI guy at Nvidia saying they were using AI for chip layout. Presumably not LLMs and I’m not going back on X to find the tweet, but just to say I think they are doing this (at least experimentally).

Re: Nvidia Trains LLM on Chip Design

#38

Post by atx_ml_guy »

xnx wrote: post #7

Google has been using machine learning for chip design since at least 2021: https://www.nature.com/articles/s41586-021-03544-w Hasn't brought about the singularity yet.

While I have no doubt that Google is working on machine learning applications for chip design, there have been a number of concerns raised with that paper:

https://retractionwatch.com/2023/09/26/nature-flags-doubts-o...

Re: Nvidia Trains LLM on Chip Design

#39

Post by omneity »

falcor84 wrote: post #22

"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind... Thus the first ultraintellig…

Keeping in mind, this holds true in a runaway fashion if the only bottleneck to more intelligence is further intelligence.

I suspect physical limitations similar to how many runaway processes in the universe are more logistical than exponential in nature.

Re: Nvidia Trains LLM on Chip Design

#40

Post by moffkalast »

omneity wrote: post #39
falcor84 wrote: post #22

"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind... Thus the first ultraintellig…

Keeping in mind, this holds true in a runaway fashion if the only bottleneck to more intelligence is further intelligence. I suspect physical limitations similar to how many runaway processes in the universe are more logistical than exponential in nature.

Logistics tend to be improvable with more inteligence though, no?

There is precedence for superhuman inteligence if you look at the best historical polymaths, and that's just what one can do with 20 W of energy. We're probably nowhere close to the universal inteligence cap in terms of physical limitations, if there even is such a thing.

Post reply on HN