Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Personal AI Factory

john-rush.com

Post reply on HN
Print view View original

141–150 of 167 posts

Re: Building a Personal AI Factory

#141

Post by simonw »

1dom wrote: post #100
simonw wrote: post #53

Earlier quoted context omitted.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM p…

> Non-trivial examples are things that would take a team of different specialist skillsets months to create.

Thank you for providing a spelled out definition of "non-trivial" there!

Re: Building a Personal AI Factory

#143

Post by simonw »

sroussey wrote: post #109
simonw wrote: post #53

Earlier quoted context omitted.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

Convert react-stockcharts to react v19. I’ve tried Claude Code and Cursor but only ended up with hilariously bad results.

I had great success with o4-mini via ChatGPT for they kind of upgrade, since of can use its search tool to look up what's changed.

I used this prompt a few weeks ago:

> This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it.

https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...

Re: Building a Personal AI Factory

#144

Post by simonw »

1dom wrote: post #108
sokoloff wrote: post #101

Earlier quoted context omitted.

> ultimately it's just doing a day job that millions of people can do sleep deprived and hungover. Doing for < $10 and under an hour what could be done in a few weeks by $10K+ worth of senior staff time is pretty valuable.

If it's something a single senior staff member can do, then - personally - I'd consider it not complex, it's relatively trivial: it can be done by literally a single person. I'm pro AI, I'm not saying it's not valuable for trivial things. But that's a distinct discussion to the trivial nature of many LLM examples/demos in relation to genuinely complex computer systems.

Maybe the definition of "non-trivial" in these conversations should be defined as "stuff an LLM system can't do yet".

Re: Building a Personal AI Factory

#145

Post by pydry »

simonw wrote: post #138
pydry wrote: post #134

Earlier quoted context omitted.

Really? This paper cut through the same kind of bullshit with puzzles: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... What do you think is so difficult about doing the same thing with coding problems?

I don't understand the connection between that paper and my comment.

They created an environment to expose LLMs to problems and test their performance which were immune from benchmark hacking using puzzles.

Your comment was about how this was unreasonably hard (for coding challenges).

Anecdotally Ive seen LLMs do all sorts of amazing shit which was obviously drawn from their training set and fall flat on their faces doing simple coding tasks which are novel enough to not appear in the training set.

Re: Building a Personal AI Factory

#146

Post by simonw »

pydry wrote: post #145
simonw wrote: post #138

Earlier quoted context omitted.

I don't understand the connection between that paper and my comment.

They created an environment to expose LLMs to problems and test their performance which were immune from benchmark hacking using puzzles. Your comment was about how this was unreasonably hard (for coding challenges). Anecdotally Ive seen LLMs do all sorts of amazing shit which was obviously drawn from their training set and fall flat on their faces doing simple coding tasks which are novel enough to not appear in the…

That Apple paper mainly demonstrated that "reasoning" LLMs - with no access to additional tools - can't solve problems that deliberately exceed their token context length.

I don't think it has much relevance at all to a conversational about how good LLMs are at solving programming problems by running tools in a loop.

I keep seeing this idea that LLMs can't handle problems that aren't in their training data and it's frustrating because anyone who has spent significant time working with these systems knows that it obviously isn't true.

Re: Building a Personal AI Factory

#147

Post by fho »

1dom wrote: post #100
simonw wrote: post #53

Earlier quoted context omitted.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM p…

Point in case: i've been trying for weeks now to generate a CFD solver that is more than the basic FDM "toy example".

The models clearly know the equations, but run into the same issues I had when implementing it myself (namely exploding simulations that the models try to paper over by applying more and more relaxation terms).

Re: Building a Personal AI Factory

#148

Post by dust42 »

simonw wrote: post #53
low_common wrote: post #46

Earlier quoted context omitted.

That's a pretty trivial example for one of these IDEs to knock out. Assembly is certainly in their training sets, and obviously docker is too. I've watched cursor absolutely run amok when I let it play around in some of my codebase. I'm bullish it'll get there sooner rather than later, but we're not there yet.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

I have one for you: implement gemma 3n multimodel support in llama.cpp

Re: Building a Personal AI Factory

#149

Post by marviel »

_1tem wrote: post #91
marviel wrote: post #20

Thanks for the writeup! I talked about a similar, but slightly simpler workflow in my post on "Vibe Specs". https://lukebechtel.com/blog/vibe-speccing I use these rules in all my codebases now. They essentially cause the AI to do two things differently: (1) ask me questions first (2) Create a `spec.md` doc, before writing any code. Seems not too dissimilar from yours, but I limit it to a single LLM

Claude Code now handles this natively with “plan mode”. Bit slow and annoying to do it manually with .md files in my opinion.

Yeah just learned about this!

The md files are actually pretty great for shareability, versioning, and picking up where you left off.

Re: Building a Personal AI Factory

#150

Post by 1dom »

simonw wrote: post #141
1dom wrote: post #100

Earlier quoted context omitted.

I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM p…

> Non-trivial examples are things that would take a team of different specialist skillsets months to create. Thank you for providing a spelled out definition of "non-trivial" there!

Haha, it was made up on the spot, thank you though! I think your articles and notes are proof that there's a lot of value and use in "trivial" examples. They're very close to the sort of examples a lot of tech people can actually use as individual professional engineers.

I think the void where non-trivial examples should be is the same space where contrarians and the last remaining few LLMs-are-useless crowd hangout.

Post reply on HN