Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing Generative AI for Circuit Board Design

blog.jitx.com

Post reply on HN
Print view View original

11–20 of 180 posts

Re: Testing Generative AI for Circuit Board Design

#11

Post by r2_pilot »

imperio59 wrote: post #4
stavros wrote: post #3

Earlier quoted context omitted.

Opus is better than GPT-4? I've heard mixed experiences.

That's because the sample size is probably small and for niche prompts or topics. It's very hard to evaluate whether a model is better than another, especially doing it in a scientifically sound way is time consuming and hard. This is why I find these types of comments like "model X is so much better than model Y" to be about as useful as "chocolate ice cream is so much better than vanilla"

And both flavors have a base flavor of excrement... Still, since I started using Claude 3 Opus (and now 3.5 Sonnet) a couple of months back, I don't see myself switching from them nor stopping use of LLM-based AI tech; it's just made me feel like the computer is actually working for and with me and even that alone can be enough to get me motivated and accomplish what I set out to do.

Re: Testing Generative AI for Circuit Board Design

#12

Post by cjk2 »

shrimp_emoji wrote: post #9
cjk2 wrote: post #7

Ex EE here > The AI generated circuit was three times the cost and size of the design created by that expert engineer at TI. It is also missing many of the necessary connections. Exactly what I expected. Edit: to clarify this is even below the expectations of a junior EE who had a heavy weekend on the vodka.

It's like a generated image with an eye missing but for circuits. :D

AI proceeds to use 2n3904 as a thyristor.

AI happy as it worked the first 10ns of the cycle.

Re: Testing Generative AI for Circuit Board Design

#13

Post by CamperBob2 »

DHaldane wrote: post #10
DHaldane wrote: post #5

Earlier quoted context omitted.

That's an interesting question - I'll take a few pokes at it now to see if there's improvement.

Update: Sonnet 3.5 is better than any other model for the circuit design and part finding tasks. Going to iterate a bit on the prompts to see how much I can push the new model on performance. Figures that any article written on LLM limits is immediately out of date. I'll write an update piece to summarize new findings.

That name threw me for a loop. 'Sonnet' already means something to EEs ( https://www.sonnetsoftware.com/ ).

Re: Testing Generative AI for Circuit Board Design

#14

Post by HanClinto »

This feels like an excellent demonstration of the limitation of zero-shot LLMs. It feels like the wrong way to approach this.

I'm no expert in the matter, but for "holistic" things (where there are a lot of cross-connections and inter-dependencies) it feels like a diffusion-based generative structure would be better-suited than next-token-prediction. I've felt this way about poetry-generation, and I feel like it might apply in these sorts of cases as well.

Additionally, this is a highly-specialized field. From the conclusion of the article:

> Overall we have some promising directions. Using LLMs for circuit board design looks a lot like using them for other complex tasks. They work well for pulling concrete data out of human-shaped data sources, they can do slightly more difficult tasks if they can solve that task by writing code, but eventually their capabilities break down in domains too far out of the training distribution.

> We only tested the frontier models in this work, but I predict similar results from the open-source Llama or Mistral models. Some fine tuning on netlist creation would likely make the generation capabilities more useful.

I agree with the authors here.

While it's nice to imagine that AGI would be able to generalize skills to work competently in domain-specific tasks, I think this shows very clearly that we're not there yet, and if one wants to use LLMs in such an area, one would need to fine-tune for it. Would like to see round 2 of this made using a fine-tuning approach.

Re: Testing Generative AI for Circuit Board Design

#15

Post by skapadia »

r2_pilot wrote: post #11
imperio59 wrote: post #4

Earlier quoted context omitted.

That's because the sample size is probably small and for niche prompts or topics. It's very hard to evaluate whether a model is better than another, especially doing it in a scientifically sound way is time consuming and hard. This is why I find these types of comments like "model X is so much better than model Y" to be about as useful as "chocolate ice cream is so much better than vanilla"

And both flavors have a base flavor of excrement... Still, since I started using Claude 3 Opus (and now 3.5 Sonnet) a couple of months back, I don't see myself switching from them nor stopping use of LLM-based AI tech; it's just made me feel like the computer is actually working for and with me and even that alone can be enough to get me motivated and accomplish what I set out to do.

"it's just made me feel like the computer is actually working for and with me and even that alone can be enough to get me motivated and accomplish what I set out to do."

This is a great way to describe what I've been feeling / experiencing as well.

Re: Testing Generative AI for Circuit Board Design

#16

Post by stavros »

imperio59 wrote: post #4
stavros wrote: post #3

Earlier quoted context omitted.

Opus is better than GPT-4? I've heard mixed experiences.

That's because the sample size is probably small and for niche prompts or topics. It's very hard to evaluate whether a model is better than another, especially doing it in a scientifically sound way is time consuming and hard. This is why I find these types of comments like "model X is so much better than model Y" to be about as useful as "chocolate ice cream is so much better than vanilla"

True, I just tried it for generating a book summary, and Sonnet 3.5 was very bad. GPT-4o is equally bad at that , gpt-4-turbo is great.

Re: Testing Generative AI for Circuit Board Design

#17

Post by utkuumur »

dindobre wrote: post #8

Using neural networks to solve combinatorial or discrete problems is a waste of time imo, but I'd be more than happy if somebody could convince me of the opposite.

There are recent papers based on diffusion that perform quite well. Here's an example of a recent paper https://arxiv.org/pdf/2406.01661. I am also working on ML-based CO. My approach has a close 1% gap on hard instances with 800-1200 nodes and less than 0.1% for 200-300 nodes on Maximum Cut, Minimum Independent Set, and Maximum Clique problems. I think these are very promising times for neural network-based discrete optimization.

Re: Testing Generative AI for Circuit Board Design

#18

Post by DHaldane »

HanClinto wrote: post #14

This feels like an excellent demonstration of the limitation of zero-shot LLMs. It feels like the wrong way to approach this. I'm no expert in the matter, but for "holistic" things (where there are a lot of cross-connections and inter-dependencies) it feels like a diffusion-based generative structure would be better-suited than next-token-prediction. I've felt this way about poetry-generation, and I feel like it migh…

My gut agrees with you that LLMs shouldn't do this well on a specialty domain.

But I think there's also the bitter lesson to be learned here: many times people say LLMs won't do well on a task, they are often surprised either immediately or a few months later.

Overall not sure what to expect, but fine tuning experiments would be interesting regardless.

Re: Testing Generative AI for Circuit Board Design

#19

Post by FourierEnvy »

cjk2 wrote: post #7

Ex EE here > The AI generated circuit was three times the cost and size of the design created by that expert engineer at TI. It is also missing many of the necessary connections. Exactly what I expected. Edit: to clarify this is even below the expectations of a junior EE who had a heavy weekend on the vodka.

Why do people think inserting an LLM into the mix will make it better than just an evolutionary or reinforcement model applied? Who cares if you can talk to it like a human?

Re: Testing Generative AI for Circuit Board Design

#20

Post by cjk2 »

DHaldane wrote: post #18
HanClinto wrote: post #14

This feels like an excellent demonstration of the limitation of zero-shot LLMs. It feels like the wrong way to approach this. I'm no expert in the matter, but for "holistic" things (where there are a lot of cross-connections and inter-dependencies) it feels like a diffusion-based generative structure would be better-suited than next-token-prediction. I've felt this way about poetry-generation, and I feel like it migh…

My gut agrees with you that LLMs shouldn't do this well on a specialty domain. But I think there's also the bitter lesson to be learned here: many times people say LLMs won't do well on a task, they are often surprised either immediately or a few months later. Overall not sure what to expect, but fine tuning experiments would be interesting regardless.

I doubt it'd work any better. Most of EE time I have spent is swearing at stuff that looked like it'd work on paper but didn't due to various nuances.

I have my own library of nuances but how would you even fine tune anything to understand the black box abstraction of an IC to work out if a nuance applies or not between it and a load or what a transmission line or edge would look like between the IC and the load?

This is where understanding trumps generative AI instantly.

Post reply on HN