Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

81–90 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#81
post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> microservices written from scratch and interacting via cleanly defined APIs.

Introducing network calls because why? How about just factoring a monolith appropriately?

Re: Test-driven development with an LLM for fun and profit

#82

Earlier quoted context omitted.

You're going to fall behind eventually, if you continue to treat LLMs with this level of skepticism, as others won't, and the output is accurate enough that it can be useful to improve the efficiency of work in a great many situations. Rarely are day-to-day written documents (e.g. an email asking for clarification on an issue or to schedule an appointment) of such importance that the occasional error is unforgivable.…

You’re narrowly addressing LLM use cases & omitting the most problematic one - LLMs as search engine replacements.

That's the opposite of problematic, that's where an LLM shines. And before you say hallucination, when was the last time you didn't click the link in a Google search result? It's user error if you don't follow up with additional validation, exactly as you would with Google. With GenAI it's simply easier to craft specific queries.

Re: Test-driven development with an LLM for fun and profit

#83

> For best results, our project structure needs to be set up with LLM workflows in mind. Specifically, we should carefully manage and keep the cognitive load required to understand and contribute code to a project at a minimum. What's the main barrier to doing this all the time? Sounds like a good practice in general.

> What's the main barrier to doing this all the time? Sounds like a good practice in general.

Misunderstanding of what "cognitive load" is. It's not measured in ability for a junior picked off a street at random to understand code they never saw before.

There are at least two components to cognitive load: knowledge and working memory. Human working memory is limited, meaning we can only keep track of so much code in our head at the same time, which sets an upper bound on code complexity we're able to handle. If the problem's inherent complexity is greater than that upper bound, you won't be able to effectively solve it at all.

The point of learning about the domain, learning design patterns, advanced programming techniques, the point of developing languages with powerful features, is to allow people to spend cognitive work up front, learning these techniques and tools, and then forever be able to handle more complexity within their limited working memory. The current zeitgeist of writing dumbest possible code anyone can understand, gets it exactly backwards: the dumber, more junior-friendly the code, the more working memory it takes up, for a fixed amount of problem complexity being addressed. In other words, the dumber code you insist on, the fewer and simpler problems you can solve with it before you hit the hard limits of human brains.

My own belief is that we're already witnessing it all the time - software increasingly sucks and is bug-ridden because industry is trying to save money by making the least experienced people available do most of the actual coding work. Basically, software quality and productivity, as well as the complexity of problems it can address, is being limited by working memory of junior developers.

(The industry doesn't actually stop people from learning - it just forces people who reached a minimum amount of competence and want to earn more money to switch to faux-management, which "senior" and above roles increasingly are.)

Re: Test-driven development with an LLM for fun and profit

#84
In my experience, I let the LLM help me produce code and tests. Most of my human effort is dedicated to verifying the tests, and then using the tests to verify the code.

Automation doesn't seem like a good idea. I feel it's mandatory to carefully guard the LLM, not only to verify that the LLM-generated tests (functions) are as expected, but also to modify some code that, while not affecting the correctness of the function, has low performance or poor readability.

Re: Test-driven development with an LLM for fun and profit

#85

Earlier quoted context omitted.

If human software engineers refused to work unless those conditions were met, what a wonderful world it would be.

They do implicitly: you can only be accidentally productive without those preconditions.

Explicitly failing fast is needed I think.

Failing slowly and producing months of unhappy results is too slow of a feedback cycle.

Re: Test-driven development with an LLM for fun and profit

#86
post #8
post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs. Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.

Indeed; I think there's a strong possibility that there's certain architectural choices where LLMs can do very well, and others where they would struggle.

There are with humans, but it's inconsistent; personally I really dislike VIPER, yet I've never felt the pain others insist comes with too much in a ViewController.

Re: Test-driven development with an LLM for fun and profit

#87
post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

> 3. If a problem can be framed in a way that a coding agent can solve... This reminds me of the South Park underwear gnomes. You picked a tool and set an expectation, then just kind of hand wave over the hard part in the middle, as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem. Does it sometimes take 50x effort to understand a problem and the agent well eno…

> as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem

It is imminently solvable! All that is necessary is to use a subset of language easier for the machine to understand and use in a very defined way; we could call this "coding language" or something similar. Even build tools to ensure we write this correctly (to avoid confusing the machine). Perhaps we could define our own algorithms using this "language" to help them along!

Re: Test-driven development with an LLM for fun and profit

#88
post #61
post #58

Earlier quoted context omitted.

It's the backwards reasoning that really frustrates me when using LLMs. You ask a question, it says sure do these things, they don't work out and you ask the LLM why not, and it replies yes that thing I told you to do wouldn't work because of these clear reasons. It would be nice to start at the end of that chain of reasoning instead of the other side. Another regular example is when it "invents" functions or classes…

> I am finding LLMs far more useful for soft skill topics than engineering type work, simply because of how often it leads me down a path that is eventually a dead end, because of some small detail that was wrong at the very beginning. Yeah I felt the same way in the beginning which is why I ended up writing my own chat app. What I've found while developing my spelling and grammar checker is that it is very unlikely…

I'm imagining a sort of tree-like or graph like interface where each subsequent prompting by the user shows each LLM's answers below it connected by lines

Re: Test-driven development with an LLM for fun and profit

#90
post #4

One trend I've noticed, framed as a logical deduction: 1. Coding assistants based on o1 and Sonnet are pretty great at coding with 2. Coding agents do massively better when they have a test-driven reward signal. 3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant. 4. From (1)-(3), if you can get all the necessary context…

(5) does not follow. We could use a strongly typed monolith with a purely functional core
Post reply on HN