Live data from Hacker News

YC is wrong about LLMs for chip design

zach.be

261–270 of 283 posts

Re: YC is wrong about LLMs for chip design

#261

Earlier quoted context omitted.

It would be hard to find any experts that could be paid "to create a private training set for fine tuning". The reason is that those experts do not own the code that they have written. The code is owned by big companies like NVIDIA, AMD, Intel, Samsung and so on. It is unlikely that these companies would be willing to provide the code for training, except for some custom LLM to be used internally by them, in which ca…

When I say “pay to create” I generally mean authoring new material, distilling your career’s expertise. Not my field of expertise but there seem to be experts founding startups etc in the ASIC space, and Bitcoin miners were designed and built without any of the big companies participating. So I’m not following why we need Intel to be involved. An obvious way to set up the flywheel here is to hire experts to do profes…

I doubt any real expert would be tempted by an offer to author new material, because that cannot be done in a good way.

One could author some projects that can be implemented in FPGAs, but those do not provide good training material for generating code that could be used to implement a project in an ASIC, because the constraints of the design are very different.

Designing an ASIC is a year-long process and it is never completed before testing some prototypes, whose manufacture may cost millions. Authoring some Verilog or VHDL code for an imaginary product that cannot be tested on real hardware prototypes could result only in garbage training material, like the code of a program that has never been tested to see if it actually works as intended.

Learning to design an ASIC is not very difficult for a human, because a human does not need a huge number of examples, like ML/AI. Humans learn the rules and a few examples are enough for them. I have worked in a few companies at designing ASICs. While those companies had some internal training courses for their designers, those courses only taught their design methodologies, but with practically no code examples from older projects, so very unlikely to how a LLM would have to be trained.

Re: YC is wrong about LLMs for chip design

#262
post #175

Earlier quoted context omitted.

I didn't get into this in the article, but one of the major challenges with achieving superhuman performance on Verilog is the lack of high-quality training data. Most professional-quality Verilog is closed source, so LLMs are generally much worse at writing Verilog than, say, Python. And even still, LLMs are pretty bad at Python!

I would imagine it is a reasonably straightforward thing to create a simulator that generates arbitrary chip designs and the corresponding verilog that can be used as training data. It would be much like how AlphaFold was trained. The chip designs don't need to be good, or even useful, they just need to be valid so the LLM can learn the underlying relationships.

I have never heard of any company, no matter how big and experienced, where it is possible to decide that an ASIC design is valid by any other means except by paying for a set of masks to be made and for some prototypes to be manufactured, then tested in the lab.

This validation costs millions, which is why it is hard to enter this field, even as a fabless designer.

Many design errors are not caught even during hardware testing, but only after mass production, like the ugly MONITOR/MWAIT bug of Intel Lunar Lake.

Randomly-generated HDL code, even if it does not have syntax errors, and even if some testbench for it does not identify deviations from its specification, is not more likely to be valid when implemented in hardware, than the proverbial output of a typewriting monkey.

Re: YC is wrong about LLMs for chip design

#263
post #240

Anything that requires deep “understanding” or novel invention is not a job for a statistical word regurgitator. I’ve yet to see a single example, in any field, of an LLM actually inventing something truly novel (as judged by the experts in that space). Where LLMs shine is in producing boilerplate -- though that is super useful. So far I have yet to see anything resembling an original “thought” from an LLM (and I use…

Define novel

Re: YC is wrong about LLMs for chip design

#264

LLMs have a long way to go in the world of EDA. A few months ago I saw a post on LinkedIn where someone fed the leading LLMs a counter-intuitively drawn circuit with 3 capacitors in parallel and asked what the total capacitance was. Not a single one got it correct - not only did they say the caps were in series (they were not) it even got the series capacitance calculations wrong. I couldn’t believe they whiffed it a…

I would expect an LLM's internal modeling to be on approximately the level of "this is a diagram of a capacitor circuit for some student's homework; electrical component calculations for homework tend to use the adding-in-reciprocal rule, because simple addition would be too straightforward for homework". > “they didn’t even teach me this in 4 years of Electrical Engineering!” (Really? They don’t teach how capacitors…

>I can absolutely believe that such people didn't learn and remember anything meaningful from those courses. Whether the course is to blame, is far more questionable.

It's the same as all the people who say "Why didn't high school teach me how to balance a check book or calculate a mortgage or blah blah?"

In nearly every case, they literally did, but you weren't paying attention.

You also had to cheat off me to pass biology, so I'm going to go ahead and press X to doubt that you "understand the immune system"

We are surrounded by people who failed to invest in their own education, and instead of facing that awful reality, they INSIST that WE are the dumb ones.

It's infuriating.

Re: YC is wrong about LLMs for chip design

#265
post #142

Earlier quoted context omitted.

> So an LLM would potentially be good at writing a first draft of that script, which Dave could then proofread/edit Right, and there’s nothing fundamentally wrong with this, nor is it a novel method. We’ve been joking about copying code from stack overflow for ages, but at least we didn’t pretend that it’s the peak of human achievement. Ask a teacher the difference between writing an essay and proofreading it. Look,…

> We’ve been joking about copying code from stack overflow for ages Tangent for a slight pet peeve of mine: "We" did joke about this, but probably because most of our jobs are not in chip design. "We" also know the limits of this approach. The fact that Stack Overflow is the most SEO optimised result for "how to center div" (which we always forget how to do) doesn't have any bearing on the times when we have an actua…

Also the Stack Overflow thing has more to do with all of us being generalists, not incompetent.

I look up "how do I sort a list in language X" because I know from school that there IS a defined good way to do it, probably built into the language, and it will be extremely idiomatic, but I haven't used language X in five years and the specifics might have changed and I don't remember the specific punctuation.

Re: YC is wrong about LLMs for chip design

#266
post #240

Anything that requires deep “understanding” or novel invention is not a job for a statistical word regurgitator. I’ve yet to see a single example, in any field, of an LLM actually inventing something truly novel (as judged by the experts in that space). Where LLMs shine is in producing boilerplate -- though that is super useful. So far I have yet to see anything resembling an original “thought” from an LLM (and I use…

There are many LLMs that are producing original "thought".

ESM3: https://www.evolutionaryscale.ai/blog/esm3-release

AlphaProof/AlphaGeometry2: https://deepmind.google/discover/blog/ai-solves-imo-problems...

MatPilot discovering new materials: https://arxiv.org/abs/2411.08063

Then of course NVidia Omniverse with their digital-twin learning.

https://blog.google/technology/ai/google-ai-big-scientific-b...

Re: YC is wrong about LLMs for chip design

#267

LLMs have a long way to go in the world of EDA. A few months ago I saw a post on LinkedIn where someone fed the leading LLMs a counter-intuitively drawn circuit with 3 capacitors in parallel and asked what the total capacitance was. Not a single one got it correct - not only did they say the caps were in series (they were not) it even got the series capacitance calculations wrong. I couldn’t believe they whiffed it a…

I would expect an LLM's internal modeling to be on approximately the level of "this is a diagram of a capacitor circuit for some student's homework; electrical component calculations for homework tend to use the adding-in-reciprocal rule, because simple addition would be too straightforward for homework". > “they didn’t even teach me this in 4 years of Electrical Engineering!” (Really? They don’t teach how capacitors…

As an educationer at the academic level the number of times I have to explain absolute basic "everybody should have learned it in school"-physics is staggering.

Re: YC is wrong about LLMs for chip design

#268
post #266
post #240

Anything that requires deep “understanding” or novel invention is not a job for a statistical word regurgitator. I’ve yet to see a single example, in any field, of an LLM actually inventing something truly novel (as judged by the experts in that space). Where LLMs shine is in producing boilerplate -- though that is super useful. So far I have yet to see anything resembling an original “thought” from an LLM (and I use…

There are many LLMs that are producing original "thought". ESM3: https://www.evolutionaryscale.ai/blog/esm3-release AlphaProof/AlphaGeometry2: https://deepmind.google/discover/blog/ai-solves-imo-problems... MatPilot discovering new materials: https://arxiv.org/abs/2411.08063 Then of course NVidia Omniverse with their digital-twin learning. https://blog.google/technology/ai/google-ai-big-scientific-b...

Taking a quick glance at all of these, they seem to be aspirational or a “brute force” type of search, which computers have always been good at, before AI. Does not seem like any novel research to me. The parameters and methods are set by humans and these systems search within a well defined space.

Re: YC is wrong about LLMs for chip design

#269

Earlier quoted context omitted.

I would expect an LLM's internal modeling to be on approximately the level of "this is a diagram of a capacitor circuit for some student's homework; electrical component calculations for homework tend to use the adding-in-reciprocal rule, because simple addition would be too straightforward for homework". > “they didn’t even teach me this in 4 years of Electrical Engineering!” (Really? They don’t teach how capacitors…

>I can absolutely believe that such people didn't learn and remember anything meaningful from those courses. Whether the course is to blame, is far more questionable. It's the same as all the people who say "Why didn't high school teach me how to balance a check book or calculate a mortgage or blah blah?" In nearly every case, they literally did , but you weren't paying attention. You also had to cheat off me to pass…

I keep thinking of a science fiction scenario of being abducted by aliens and then being rescued by alien cops.

“Where are you from?”

“What’s the chemistry of your required sustenance?”

“How long is your sleep cycle as measured with physical time constants?”

And similar basic questions could not be answered by 99.9% of the human population.

Fundamentally, almost none of us can give an accurate answer to what were made of, where we’re from, or what we need to survive.

Re: YC is wrong about LLMs for chip design

#270
post #175

Earlier quoted context omitted.

I would imagine it is a reasonably straightforward thing to create a simulator that generates arbitrary chip designs and the corresponding verilog that can be used as training data. It would be much like how AlphaFold was trained. The chip designs don't need to be good, or even useful, they just need to be valid so the LLM can learn the underlying relationships.

I have never heard of any company, no matter how big and experienced, where it is possible to decide that an ASIC design is valid by any other means except by paying for a set of masks to be made and for some prototypes to be manufactured, then tested in the lab. This validation costs millions, which is why it is hard to enter this field, even as a fabless designer. Many design errors are not caught even during hardw…

Validating an arbitrary design is hard. It's equivalent to the halting problem. Working backwards using specific rules that guarantee validity is much easier. Again, the point is not to produce useful designs. The generated model doesn't need to be perfect, indeed it can't be, it just needs to be able to avoid the same issues that humans are looking for.
Post reply on HN