Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's the largest software project AI can complete on its own?

epoch.ai

Post reply on HN
Print view View original

21–30 of 110 posts

Re: What's the largest software project AI can complete on its own?

#21

Post by rco8786 »

Jdstanhope wrote: post #11

I have been using Claude to build a clone of Bash in Rust ( https://github.com/jdstanhope/huck ). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 te…

> it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

you ever worked with human engineers before? :D

Re: What's the largest software project AI can complete on its own?

#22

Post by justincormack »

Jdstanhope wrote: post #11

I have been using Claude to build a clone of Bash in Rust ( https://github.com/jdstanhope/huck ). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 te…

I find most of the time it would be fine on its own, but every now and again you avoid a really bad degradation through guidance. Some kinds of big change definitely problematic.

Re: What's the largest software project AI can complete on its own?

#23

Post by logicchains »

stared wrote: post #18

I am curious how Claude Opus 5 fares - similar, better, or (my guess) worse than Fable 5.

For really hard problems (maths) it's far far worse than Fable 5, not even as good as Sol 4.6. Current benchmarks are inadequate as they fail to capture how much worse Opus 5 is than Fable 5.

Re: What's the largest software project AI can complete on its own?

#24

Post by Jdstanhope »

rco8786 wrote: post #21
Jdstanhope wrote: post #11

I have been using Claude to build a clone of Bash in Rust ( https://github.com/jdstanhope/huck ). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 te…

> it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests. you ever worked with human engineers before? :D

I was going to say that it's pretty much exactly like working with an experienced software engineer ... except it never seems to take a step back and look at the bigger picture. It will happily fix the same class of bugs over and over without realizing a bigger change is required.

Re: What's the largest software project AI can complete on its own?

#25

Post by muglug »

> MirrorCode only covers programs with plain-text output. MirrorCode does not test AI abilities in software that is inherently visual

It’s an interesting eval but this is a distinct limitation — most applications that consumers use directly are not purely text-based.

Re: What's the largest software project AI can complete on its own?

#26

Post by zsoltkacsandi »

logicchains wrote: post #23
stared wrote: post #18

I am curious how Claude Opus 5 fares - similar, better, or (my guess) worse than Fable 5.

For really hard problems (maths) it's far far worse than Fable 5, not even as good as Sol 4.6. Current benchmarks are inadequate as they fail to capture how much worse Opus 5 is than Fable 5.

I agree, Opus 5 is terrible. Worse than Opus 4.8.

Re: What's the largest software project AI can complete on its own?

#27

Post by AlotOfReading »

marginalia_nu wrote: post #14
thisisauserid wrote: post #12

I believe they can make ANY software project the largest it can be! This solution should look for more problems.

Yeah came here to say the same thing. The Hard Problem of software engineering isn't making a complex monstrosity with millions of lines of code and endless feature creep, but not doing that. This was as true 20 years as it is today.

I apologize for such a big program — I didn't have enough tokens to write a small one.

- my coworkers

Re: What's the largest software project AI can complete on its own?

#28

Post by swframe2 »

JauntyHatAngle wrote: post #16
swframe2 wrote: post #9

Running a Loop on the well defined goal seems promising: https://www.youtube.com/shorts/EtxdB7gpFqY

I'm a bit suspicious of this as there are open source excel clones already. It's impressive AI can put together it's own version, but I can't help but feel it's probably just relying on existing excel clones as it's a well trodden space. Novel applications are what impress me. Not a clone.

Agreed. Novel would be very impressive. AI tech is struggling with that.

It is important not to wait for today's AI tech to get to AGI. It is better to exploit the useful things it can do today while waiting for it to get better tomorrow.

Cloning existing software might be useful today. It allows the model to

1) learn from its mistakes

2) mix known solutions (embed product A inside product B)

3) mine value (turn product A into a set of reusable libraries)

etc.

Re: What's the largest software project AI can complete on its own?

#29

Post by linsomniac »

thisisauserid wrote: post #12

I believe they can make ANY software project the largest it can be! This solution should look for more problems.

You say that, but I asked Codex to build a Asteroids (but FPS) as a web app, and it wrote it in under 4K LOC, and a Tempest implementation in 6.5K LOC. Those seem like really reasonable line counts to me.

I also built a caching proxy server for apt packages, that's a bigger project, and that is 30.5K lines of golang, which doesn't seem bad compared to apt-cacher-ng which is 19.3K of C++ (my project does significantly more).

Post reply on HN