Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's the largest software project AI can complete on its own?

epoch.ai

Post reply on HN
Print view View original

101–110 of 110 posts

Re: What's the largest software project AI can complete on its own?

#101

Post by harrouet »

sreekanth850 wrote: post #39

We (Team of 3) built a retrieval engine from scratch in .NET for investigative search, with AI doing most of the backend implementation while humans handled the design and architecture and front end. It was developed using a task based, incremental approach. Includes more than 3,000 tests and extensive documentation, and is optimized for asynchronous ingestion using NATS and a disk backed WAL. It includes a custom WA…

"we have more than 3,000 tests"...

Great, but since no human has looked at them, how do you know that they are relevant and actually test undesired behavior?

Also, would you be able to point at which tests should be added in the future?

Re: What's the largest software project AI can complete on its own?

#102

Post by sreekanth850 »

harrouet wrote: post #101
sreekanth850 wrote: post #39

We (Team of 3) built a retrieval engine from scratch in .NET for investigative search, with AI doing most of the backend implementation while humans handled the design and architecture and front end. It was developed using a task based, incremental approach. Includes more than 3,000 tests and extensive documentation, and is optimized for asynchronous ingestion using NATS and a disk backed WAL. It includes a custom WA…

"we have more than 3,000 tests"... Great, but since no human has looked at them, how do you know that they are relevant and actually test undesired behavior? Also, would you be able to point at which tests should be added in the future?

The tests are based on actual expected outputs, not generated only to increase coverage. For every feature, we have a phase in TASK Ledger with scope, objective, goals and outcome. And the final part will be test coverage with expectations. I cannot claim that a human has reviewed every test line by line. However, humans defined and reviewed the expected behaviour, architecture, invariants, and feature outcomes against which those tests were written. Also, when a bug is reported, we add a regression and prove the bug and then fix and test should pass.

We also use a headless API with a fully decoupled frontend, and the frontend is written entirely by developers. This helps catch additional issues during integration because every endpoint is exercised and validated through the actual userfacing workflows.

I may not be able to convince developers about the process we follow, but we got it working, and tested for concurrency, load and fanout behaviour. This is our primary requirement, as this is targeted towards large scale government customers. Regarding future tests, they will be added for new features, reported bugs, newly discovered failure modes, and regressions. This level of test coverage also helps us use AI without allowing significant drift in the codebase.

Re: What's the largest software project AI can complete on its own?

#103

Post by Art9681 »

ErenayDev wrote: post #42

Seems like most of developers and humans vibecoding new projects with LLMs. But I can see they have problems in the codebase with AI generated code, or security breachs etc. Whats the limit of it? In other words, I'm curious about what the future holds.

Given all of the CVE's being published for software that's been around decades and being tended to by the most talented businesses and individuals in the world, the least of my concerns are vibe coded projects that will never be used.

All software has problems. Most engineers spend their careers fixing issues caused by previous engineers. This was the case before LLMs, and it will continue to be the case for the foreseeable future.

Re: What's the largest software project AI can complete on its own?

#104

Post by StableAlkyne »

khazhoux wrote: post #32
Jdstanhope wrote: post #11

I have been using Claude to build a clone of Bash in Rust ( https://github.com/jdstanhope/huck ). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 te…

Very cool, but I'm skeptical. Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture, with "fixes" just working around broken code, and tons of duplication that lead to whack-a-mole. The system works at first, but then bugs start piling up and eventually take over. Everyone I talk to shares the same experience. Not to mention that i…

> Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture

I've noticed this pattern as well when people decide to vibe code a large project.

They usually get their MVP quickly, but as soon as they want a new feature or to fix a bug, the LLM takes the shortest possible route to a solution. The problem is that after about a dozen rounds of "the honest fix" and "load-bearing code" (insert your favorite claudeism) it's a tangled mess of enterprise-grade spaghetti.

I think it's because the people full-on vibe coding just don't have the experience or time to design large scale systems and anticipate what might cause problems if they want to change something later.

IMO coding agents can be useful, but the only safe place is the manageable changes you would design up and hand off to a junior anyway (but can't because your company decided to outsource years ago).

Re: What's the largest software project AI can complete on its own?

#105

Post by peder »

khazhoux wrote: post #63
peder wrote: post #36

Earlier quoted context omitted.

> tons of duplication that lead to whack-a-mole The future is duplication. Code is no longer meant to be read by humans but by machines.

It’s not human readability I care about. The problem with duplicate code is that the agent will fix one instance, but leave the others buggy. Hilarity ensues.

That doesn't happen with a proper test suite. Give guardrails and it's not an issue.

Re: What's the largest software project AI can complete on its own?

#106

Post by IncandescentGas »

dnnehgf wrote: post #83
eithed wrote: post #66

I'm looking at these reports and then at my sessions with Claude. In latest one, it got stuck in a loop fixing one thing and breaking another for 10 iterations, until I've noticed and gave it the insight. Or where it started continuously writing "echo", I don't know for what purpose. And I ask how, how can you not be the driver; how can you let these agents roam free and actually accomplish things.

we bomb schools and walk past beggars and devote our surplus time and money to scones and brain rot and gym memberships. our c code probably looks at us and says: how, how can you let these agents roam free. llms are not code and are incompetent and inefficient relative to code, but on the other hand you cannot roam without a little echo echo echo in your blood.

Thanks, your comment got this song stuck in my head. (I like my gym membership, thank you very much. I even visited the gym once several months ago.)

We wasted every second dime On diets, lawyers, shrinks, and apps, and flags And plastic surgery

- So long & thanks for all the fish, a perfect circle

Re: What's the largest software project AI can complete on its own?

#107

Post by micromacrofoot »

dd8601fn wrote: post #19
micromacrofoot wrote: post #7

Earlier quoted context omitted.

well if driving is supposedly safer when automated, why not code

The qualities of computation that make autopilots safer than human drivers aren’t the same qualities that produce better designed software products.

to err is human

Re: What's the largest software project AI can complete on its own?

#108

Post by micromacrofoot »

LetsGetTechnicl wrote: post #15
micromacrofoot wrote: post #7

Earlier quoted context omitted.

well if driving is supposedly safer when automated, why not code

Is it actually safer? If you can't understand the code you can't verify it's safety, and AI code often has bugs in it. You can't trust implicitly.

Same applies to driving cars

Re: What's the largest software project AI can complete on its own?

#109

Post by micromacrofoot »

ModernMech wrote: post #10
micromacrofoot wrote: post #7

Earlier quoted context omitted.

well if driving is supposedly safer when automated, why not code

It’s supposedly safer than the average driver, not safer than professional drivers.

the average american spends 60 minutes a day driving, that's thousands of hours by the time you reach 25... at what point is that not a professional level of experience?
Post reply on HN