Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test-driven development with an LLM for fun and profit

blog.yfzhou.fyi

Post reply on HN
Print view View original

41–50 of 91 posts

Re: Test-driven development with an LLM for fun and profit

#41

Post by thuuuomas »

redcobra762 wrote: post #21
smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

You're going to fall behind eventually, if you continue to treat LLMs with this level of skepticism, as others won't, and the output is accurate enough that it can be useful to improve the efficiency of work in a great many situations. Rarely are day-to-day written documents (e.g. an email asking for clarification on an issue or to schedule an appointment) of such importance that the occasional error is unforgivable.…

You’re narrowly addressing LLM use cases & omitting the most problematic one - LLMs as search engine replacements.

Re: Test-driven development with an LLM for fun and profit

#42

Post by lifeisstillgood »

ozten wrote: post #31
lifeisstillgood wrote: post #28

Earlier quoted context omitted.

Can you extend that - what was the domain, how did you start? I would like to give this a try but am not quite sure I get it?

Backend coding for web services. In the past I would hand write 8 or 9 unit tests. Now I write the first one or two and then brain dump anything else into the LLM prompt. It then outputs mine plus 6 or more. I delete any that seem low value or ridiculous or have a follow up prompt to ask for refinements. Then just copy/pasta back into the codebase out of the chat.

That simple ? I’ll try it

Re: Test-driven development with an LLM for fun and profit

#43

Post by sdesol »

horsawlarway wrote: post #34
sdesol wrote: post #13

Earlier quoted context omitted.

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "Wh…

Isn't this essentially making the point of the post above you? For comparison - if I just do a web search for "Did homer simpson go to mars" I get immediately linked to the wikipedia page for that exact episode ( https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles ), and the plot summary is less to read than your LLM output - It clearly summarizes that Marge & Lisa (note - NOT homer) almost went to mars, but did n…

> Isn't this essentially making the point of the post above you?

Yes. This is why I wrote the chat app, because I mistrust LLMs, but I do find them extremely useful when you approach them with the right mindset. If answering "Did Homer Simpson go to Mars?" correctly is critical, then you can choose to require a 100% consensus, otherwise you will need a fallback plan.

When I asked all the LLMs about the Wikipedia article, they all correctly answered "No" and talked about Marge and Lisa in the future without Homer.

Re: Test-driven development with an LLM for fun and profit

#44

Post by sdesol »

__MatrixMan__ wrote: post #30
sdesol wrote: post #14

Earlier quoted context omitted.

Agreed. If the microservice does not provide any value from being isolated, it is just a function call with extra steps.

I think the argument is that the extra value provided is a small enough context window for working with an LLM. Although I'd suggest making it a library if one can manage, that gives you the desired context reduction bounded by interfaces without taking on the complexities of adding an additional microservice. I imagine throwing a test at an LLM and saying: > hold the component under test constant (as well as the tes…

> I think the argument is that the extra value provided is a small enough context window for working with an LLM.

I'm not sure moving something that could work as function to a microservice would save much context. If anything, I think you are adding more context, since you would need to talk about the endpoint and having it route to the function that does what you need. When it is all over, you need to describe what the input and output is.

Re: Test-driven development with an LLM for fun and profit

#45

Post by sdesol »

energy123 wrote: post #35
smusamashah wrote: post #5

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

We need a hallucination benchmark. My experience is, o1 is very good at avoiding hallucinations and I trust it more, but o1-mini and 4o are awful.

Well given the price $15.00 / 1M input tokens and $60.00 / 1M output* tokens, I would hope so. Given the price, I think it is fair to say it is doing a lot of checks in the background.

Re: Test-driven development with an LLM for fun and profit

#46

Post by p1necone »

Swizec wrote: post #37
p1necone wrote: post #27

Earlier quoted context omitted.

By large scale do you mean large software or large amounts of developers? Because there's some absolutely massive software in terms of feature set, usefulness and even LoC (not that that is a useful measurement) etc out there made by very small teams. I'm not sure that you've got the causal relationship the right way around here re: architecture:team size.

What does team size have to do with this? Small teams can (and should) absolutely build modularized software ... You simply cannot build a [working/maintainable] large piece of software if everything is connected to everything and any one change may cause issues in conceptually unrelated pieces of code. As soon as your codebase is bigger than what you can fully memorize, you need modules, separation of concerns, etc.

Sure I agree with that, but microservices are just one of many ways to modularize software/achieve separation of concerns.

I assumed you were talking about team size specifically because that is the thing that a microservice architecture uniquely enables in my experience.

Re: Test-driven development with an LLM for fun and profit

#47

Post by energy123 »

sdesol wrote: post #45
energy123 wrote: post #35

Earlier quoted context omitted.

We need a hallucination benchmark. My experience is, o1 is very good at avoiding hallucinations and I trust it more, but o1-mini and 4o are awful.

Well given the price $15.00 / 1M input tokens and $60.00 / 1M output* tokens, I would hope so. Given the price, I think it is fair to say it is doing a lot of checks in the background.

It is expensive. But if I'm correct about o1, it means user mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 (or better) models as their daily driver.

Re: Test-driven development with an LLM for fun and profit

#48

Post by jacobpedd »

> For best results, our project structure needs to be set up with LLM workflows in mind. Specifically, we should carefully manage and keep the cognitive load required to understand and contribute code to a project at a minimum.

What's the main barrier to doing this all the time? Sounds like a good practice in general.

Re: Test-driven development with an LLM for fun and profit

#49

Post by krapp »

jacobpedd wrote: post #48

> For best results, our project structure needs to be set up with LLM workflows in mind. Specifically, we should carefully manage and keep the cognitive load required to understand and contribute code to a project at a minimum. What's the main barrier to doing this all the time? Sounds like a good practice in general.

It isn't good practice. You don't want people contributing to a project who don't understand the code they submit or the project they're contributing to, because you'll just need to make that up with more effort debugging garbage code. The cognitive load required to actually learn how things work is a necessary filter for minimum effort and quality.
Post reply on HN