Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Personal AI Factory

john-rush.com

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Building a Personal AI Factory

#151

Post by edmundsauto »

1dom wrote: post #100
simonw wrote: post #53

Earlier quoted context omitted.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM p…

Current state AI is a best fit for jobs that can be easily verified as correct. In my 20+ years, this is at least 75% of the work I’ve ever done. Maybe 99.999% (I have led a very boring career.)

There’s an enormous amount of value in doing this. For the harder problems you mentioned - most IC SWE are also incapable or unwilling to do the work. So maybe the current state has equivalent capabilities to 95% of coders out there? But it works faster, cheaper, and doesn’t object to tedious work like documentation. It doesn’t require labor law compliance, hiring, onboarding/offboarding, or cause interpersonal conflict.

Re: Building a Personal AI Factory

#152

Post by codemonkey-zeta »

> Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right.

In case the author is lurking, you may want to apply the same fix they do in clojure-mcp: https://github.com/bhauman/clojure-mcp/blob/8150b855282babcd...

The insight that team had was that LLMs get confused with parens, but they are excellent at indentation, so if you run parinfer over the LLMs output it will be correct in 99% of cases.

Re: Building a Personal AI Factory

#153

Post by apwell23 »

stavros wrote: post #124
petesergeant wrote: post #113

Earlier quoted context omitted.

> I haven't been able to get them to write hard code on their own, though For sure, and me neither, for what it's worth. But most of the code I write isn't "hard" code; the hard code is also the stuff I enjoy writing the most. I will note that a few months ago I found them helpful for small things inside the GPT window, and then tried agentic mode (specifically Roo, then Claude Code), and have seen a huge speedup in…

Agreed, I no longer have to write the same code for the Nth time, or spend two minutes times a hundred looking up API docs. I love it.

> write the same code for the Nth time

who does this though ? maybe you should extract that into a library/method/abstraction ?

Re: Building a Personal AI Factory

#154

Post by x0x0 »

simonw wrote: post #53
low_common wrote: post #46

Earlier quoted context omitted.

That's a pretty trivial example for one of these IDEs to knock out. Assembly is certainly in their training sets, and obviously docker is too. I've watched cursor absolutely run amok when I let it play around in some of my codebase. I'm bullish it'll get there sooner rather than later, but we're not there yet.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

I have one: features I've tried this on in my codebase. Because claude and gemini have both failed pretty badly.

So it's pretty stupid to just assume that critics haven't tried.

Example feature: send analytics events on app start triggered by notifications. Both Gemini and Claude completely failed to understand the component tree; rewrote hundreds of lines of code in broken ways; and even when prompted with the difficulty (this is happening outside of the component tree), failed to come up with a good solution. And even when deliberately prompted not to, like to simultaneously make cosmetic code changes to other pieces of the files they're touching.

Re: Building a Personal AI Factory

#155

Post by nico »

NitpickLawyer wrote: post #121
nico wrote: post #75

> If you know Factorio you know it’s all about building a factory that can produce itself This is a very interesting concept Could this be extended to the point of an LLM producing/improving itself? If not, what are the current limitations to get to that point?

> Could this be extended to the point of an LLM producing/improving itself? Check out aider writing aider stats here: https://aider.chat/HISTORY.html

Super interesting, thank you for the link

Aider writing its own code is definitely cool and within the same concept

I’d love to see an LLM or some sort of coding model that modifies/trains the model itself

Re: Building a Personal AI Factory

#156

Post by Aeolun »

manmal wrote: post #78
Aeolun wrote: post #64

Earlier quoted context omitted.

> I have two MCPs installed (playwright and context7) but it never seems like Claude decides to reach for them on its own. Not even when you add ‘memories’ that tell it to always use those tools in certain situations? My admonitions to always run repomix at the start of coding, and always run the build command before crying victory seem to be followed pretty well anyway.

What do you tell Claude to do with repomix? Get an overview into the context?

Yeah, it’s just a shortcut to it exploring the code for half an hour before doing something. At least it seems to make its searching more targeted.

Re: Building a Personal AI Factory

#157

Post by pydry »

simonw wrote: post #146
pydry wrote: post #145

Earlier quoted context omitted.

They created an environment to expose LLMs to problems and test their performance which were immune from benchmark hacking using puzzles. Your comment was about how this was unreasonably hard (for coding challenges). Anecdotally Ive seen LLMs do all sorts of amazing shit which was obviously drawn from their training set and fall flat on their faces doing simple coding tasks which are novel enough to not appear in the…

That Apple paper mainly demonstrated that "reasoning" LLMs - with no access to additional tools - can't solve problems that deliberately exceed their token context length. I don't think it has much relevance at all to a conversational about how good LLMs are at solving programming problems by running tools in a loop. I keep seeing this idea that LLMs can't handle problems that aren't in their training data and it's f…

It demonstrated that there was a hard limit on the complexity of a puzzle that LLMs could solve no matter how many tokens they threw at it (using a form of puzzle construction that it ensured that the LLM couldn't just refer to its training data to solve it).

Re: Building a Personal AI Factory

#158

Post by PhilippGille »

> Next claude code execute the plan, either with sonnet 3.7 or sonnet 4 depending on the complexity of the task. Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right.

This made me chuckle.

Perfect example of why heavily LLM-driven devs and processes might want to pick a popular programming language which the LLM had a ton of training data for. Or a strong point for specialized LLMs (e.g. here it could be a smaller/cheaper/faster Clojure-specialized model).

Re: Building a Personal AI Factory

#159

Post by puersum »

I believe we need to find more effective ways to integrate AI into our workflows. Anyone who is actively trying to adopt AI has likely encountered similar challenges, yet a definitive solution has yet to emerge. In my view, a key principle at this stage is to assign AI minimal responsibility and highly specific tasks.

For example, I'm currently experimenting with an agent workflow for stock research. I've set up two AI roles: a 'Bullish Guy' and a 'Bearish Guy' and have them debate the pros and cons of a specific stock. The premise is that through this adversarial process, the AIs are forced to research opposing viewpoints, leading to a more comprehensive understanding and a superior final analysis. The idea was inspired by the kinds of arguments you see on social media.

Re: Building a Personal AI Factory

#160

Post by kayge »

simonw wrote: post #53
low_common wrote: post #46

Earlier quoted context omitted.

That's a pretty trivial example for one of these IDEs to knock out. Assembly is certainly in their training sets, and obviously docker is too. I've watched cursor absolutely run amok when I let it play around in some of my codebase. I'm bullish it'll get there sooner rather than later, but we're not there yet.

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

The "No True Scotsware" problem? :)
Post reply on HN