Live data from Hacker News

What's the largest software project AI can complete on its own?

epoch.ai

91–100 of 110 posts

Re: What's the largest software project AI can complete on its own?

#91

- I ll consider any of these models as god tier the day one of these 2 happen - They build a browser engine from scratch that implements every whatwg spec for the last 15 yrs, fixes every bug chromium has and runs 10x faster than chromium - they build a super highly optimized version of windows that runs much much faster than windows 11

People who are 100x more productive than before should seriously consider doing one of these. Should only take a few months at 100x productivity.

Re: What's the largest software project AI can complete on its own?

#92
post #49

This actually proves the hardest part of software engineering is defining requirements.

That makes it sound like a PM problem. The hardest part is building the mental model which doesn't crumble. I have worked with humans who had the same failure modes as LLMs.

Re: What's the largest software project AI can complete on its own?

#94
post #36

Earlier quoted context omitted.

> tons of duplication that lead to whack-a-mole The future is duplication. Code is no longer meant to be read by humans but by machines.

If that's true (I very much doubt it), then have the LLMs generate something like WASM binary and run that everywhere instead. No need for human readable languages. Or if LLMs are not that good at binary format, then some textual representation optimized for LLMs instead of humans that compiles to WASM, the JVM or Rust binary. No need for multiple programming languages anymore. But again, doubt that happens.

Let them. I want to see the faces of middle management when the machine stops doing 'beep', and starts making expensive sounds instead. ;)

Re: What's the largest software project AI can complete on its own?

#96
I think the best part about this benchmark is that they finally stopped pretending you can evaluate a complex engineering task with $ 5 worth of inference. If a task takes a human weeks, you need to give the agent a comparable search space to find solutions

Re: What's the largest software project AI can complete on its own?

#97
post #21

Earlier quoted context omitted.

> it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests. you ever worked with human engineers before? :D

I was going to say that it's pretty much exactly like working with an experienced software engineer ... except it never seems to take a step back and look at the bigger picture. It will happily fix the same class of bugs over and over without realizing a bigger change is required.

That’s a direct result of how they’re fine tuned. RLHF and other mechanisms reward quick, locally correct answers that solve the user’s immediate problem. A reward for a solution like "take a two-day pause, rip out half the modules, and rewrite the core" just straight-up doesn't exist in training datasets

Re: What's the largest software project AI can complete on its own?

#98
Accurately? About a ~500 line React component or API endpoint given lots of very good context

Which is not bad

I don’t bother with the agentic stuff anymore than I would run a calculator on auto - what is the actual point if you think of what that really means

Re: What's the largest software project AI can complete on its own?

#100
post #86

Earlier quoted context omitted.

Very cool, but I'm skeptical. Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture, with "fixes" just working around broken code, and tons of duplication that lead to whack-a-mole. The system works at first, but then bugs start piling up and eventually take over. Everyone I talk to shares the same experience. Not to mention that i…

> Not to mention that if I, human overseer, don't course correct it many times a day, it goes off into Neverland. Doesn't that happen to a human engineering team if you don't course correct it many times a month (which in code output is similar to an LLM in a day)?

Can't tell if this comment is serious. No, there are individual people who are capable of writing operating systems, browsers, source control systems, complex video games, and programming languages with no external guidance, design reviews, or anything. They are fully autonomous intelligent beings with good taste, powerful programming ability, and the vision to set long term and short term goals, craft plans, and execute without outside supervision.

There are also small teams of experts who need little to no supervision, and will NOT devolve without constant correction. They will do things like look for an existing function before writing a new one, or use good judgement when trying to find a library, rather than reinventing the wheel.

Post reply on HN