Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The day I started believing in unit tests

mental-reverb.com

Post reply on HN
Print view View original

211–220 of 269 posts

Re: The day I started believing in unit tests

#211

Post by __MatrixMan__ »

qup wrote: post #177
__MatrixMan__ wrote: post #29

Earlier quoted context omitted.

You should try switching it up. Write the tests and then ask the LLM to write the code that makes them pass. I find I'm more likely to learn something in this mode.

How effective is the LLM when used this way, compared to normally?

I don't know what normally is, but I'd say it works pretty well.

Often the challenge is that the context for what you're trying to do is sprawling. There's just too many files and they're all too long: you end up exceeding the context window or filling it with 99% irrelevant stuff. Typically the structures you build for tests are smaller and more focused on the particular instance you're worried about, which I think is a better way to talk to an LLVM.

You don't have to explain, for instance, that there's data in production which doesn't match the schema in the code so it must be cautious to avoid running afoul of that difference. Instead you've mocked that data, so it's right there in the same code with the test that it's trying to make pass.

Re: The day I started believing in unit tests

#212

Post by __MatrixMan__ »

makeitdouble wrote: post #208
__MatrixMan__ wrote: post #29

Earlier quoted context omitted.

You should try switching it up. Write the tests and then ask the LLM to write the code that makes them pass. I find I'm more likely to learn something in this mode.

I'd argue having useable LLMs kind of brings out how problematic TDD is. Imagine the dumbest function you have to write: a product A and a street address as input, and the shipping cost as an output. How many test cases would you write to be absolutely sure that function actually does what you want it to do, and be confident it doesn't have weird exceptions that the LLM injected randomly ? I'd assume you'd still vet…

If it's hundreds of rambling lines then I'm not going to be able to get it past my linter anyhow (complexity thresholds), nor am I going to be able to get it past my team when they review it. So yeah, that's a problematic case, but it's one I'm going to have to refactor to avoid with or without an LLM in the loop.

Re: The day I started believing in unit tests

#213

Post by rcxdude »

m3kw9 wrote: post #86
cfiggers wrote: post #48

Earlier quoted context omitted.

> if we want to call ourselves engineers we need to hold ourselves to engineering standards. Bridge builders do not get to skip tests. Bravo. We need more of this mindset in the world, and also more collective will to encourage it in one another. YOU are the kind of engineer I want writing the code that goes in my Dad's pacemaker or the cruise control in my wife's car.

If you have worked in places where safety is critical, you wouldn’t say something so shallow. In those places they place human verification above all else. They have a thick book where you do a full run and is double checked, they don’t f around with unit tests and say this is good to go

For all the testing and paperwork, the code in safety critical applications is still frequently awful and riddled with bugs, following such a process does not actually guarantee good software, it mostly just means you need a lot of paper pushers.

Re: The day I started believing in unit tests

#214

Post by MoreQARespect »

berkes wrote: post #43
atticora wrote: post #38

Earlier quoted context omitted.

My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would…

> My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit…

>No-one said that integration tests can't also be very valuable.

Integration tests are a better kind of default test because they bring value under pretty much all circumstances.

Nobody said that unit tests cant also be valuable and under just the right circumstances i.e. - complex stateless code behind a stable API.

Unit tests shine in that environment - theyre not impeded by their crippling lack of realism because that stable abstraction walls off the rest of reality. And theyre very fast.

Most code isnt parsers, calculation engines, complex string manipulation, etc. - but when it is unit tests really do kick ass.

They just suck so badly at testing code that doesnt fit that mold. Which, to be fair, is most code. I dont write a lot of parsers at work. My job involes moving data into databases, calling APIs, linking up message queues, etc.

Re: The day I started believing in unit tests

#215

Post by gorgoiler »

feoren wrote: post #130
gorgoiler wrote: post #114

Earlier quoted context omitted.

I often think of unit tests as being programmable types, like Eiffel pre/post conditions or functional languages with types like Even and Odd. For example, in double(x) -> y you can use types to say x belongs to the set of all integers and y must also must be in that set, but that’s about all you can say in Python. Unit testing lets you express that y must be an even number with the same sign as x. It is like formal…

But you literally cannot possibly test that assertion for all x . Let's take a slightly harder problem: prove (or at least test conclusively) that for all integer x , the output y of the following function is always even: y = x^2 + x + 2 There is essentially no way to prove this for all x by simply testing all integers. If your integers are 64-bit, you don't have enough time in the lifespan of the universe. On the ot…

Could you do this in Python (using positive integers as an example of a type rather than even numbers)?:

  class N(int):
    def __init__(self, z: int):
      assert z > 0
      super().__init__(z)

  def log2(n: N) -> float:
    …

  log2(N(32))   # 5.0
  log2(32)      # type error
  log2(N(-32))  # runtime error
You are still relying on the runtime to detect errors and it’s annoying to have to cast all ints to Ns, but you at least won’t ever take the log of a negative number.

Re: The day I started believing in unit tests

#216

Post by RaftPeople »

randomdata wrote: post #168
RaftPeople wrote: post #167

Earlier quoted context omitted.

> Kent Beck, who invented the term unit test, was quite clear that a unit test is a test that exists independent of other tests Kent Beck didn't invent the term "unit test", it's been used since the 70's (at minimum). > I am not sure why you would want anything other than unit tests? The reason is to produce higher quality code than if you rely on unit tests only. Generally, unit tests catch a minority of bugs, other…

> other tests like end to end testing help catch the remainder. End-to-end tests are unit tests, generally speaking. Something end-to-end can be captured within a unit. The divide you are trying to invent doesn't exist, and, frankly, is nonsensical.

> End-to-end tests are unit tests, generally speaking.

Generally, in the software industry, those terms are not considered the same thing, they are at opposite ends of a spectrum. Unit tests are testing more isolated/individual functionality while the end to end test is testing an entire business flow.

Here's an example of one end to end test (with validations happening at each step):

1-System A sends Inventory availability to system B

2-The purchasing dept enters a PO into system B

3-System B sends the PO to system A

4-System A assigns the PO to a Distribution Center for fulfillment

5-System A fulfills the order

6-System A sends the ASN and Invoice to system B

7-System B users process the PO receipt

8-System B users perform three way match on PO, Receipt and Invoice documents

Re: The day I started believing in unit tests

#217

Post by MoreQARespect »

makeitdouble wrote: post #208
__MatrixMan__ wrote: post #29

Earlier quoted context omitted.

You should try switching it up. Write the tests and then ask the LLM to write the code that makes them pass. I find I'm more likely to learn something in this mode.

I'd argue having useable LLMs kind of brings out how problematic TDD is. Imagine the dumbest function you have to write: a product A and a street address as input, and the shipping cost as an output. How many test cases would you write to be absolutely sure that function actually does what you want it to do, and be confident it doesn't have weird exceptions that the LLM injected randomly ? I'd assume you'd still vet…

TDD works best if you default to testing at the outer shell of the app - e.g. translating a user story into steps executed by playwright against your web app and only TDDing lower layers once youve used those higher level tests to evolve a useful abstraction underneath the outer shell.

It seems to be taught in a fucked up way though where you imagine you want a car object and a banana object and you want to insert the banana into a car or some other kind of abstract nonsense.

Re: The day I started believing in unit tests

#219

Post by randomdata »

RaftPeople wrote: post #216
randomdata wrote: post #168

Earlier quoted context omitted.

> other tests like end to end testing help catch the remainder. End-to-end tests are unit tests, generally speaking. Something end-to-end can be captured within a unit. The divide you are trying to invent doesn't exist, and, frankly, is nonsensical.

> End-to-end tests are unit tests, generally speaking. Generally, in the software industry, those terms are not considered the same thing, they are at opposite ends of a spectrum. Unit tests are testing more isolated/individual functionality while the end to end test is testing an entire business flow. Here's an example of one end to end test (with validations happening at each step): 1-System A sends Inventory avail…

> Here's an example of one end to end test

Bad example, perhaps, but that's also a unit test[1]. Step 8 is dependent on the state of step 1, and everything else in between, so it cannot be reduced any further (at last not without doing stupid things). That is your minimum viable unit; the individual, isolated functionality.

[1] At least so long as you don't do something that couples it with other tests, like modifying a shared database in a way that that will leave another test in an unpredictable state. But I think we have all come to agree that you should never do that – going back to the reality that the term unit test serves no purpose anymore. For all intents and purposes, all tests now written are unit tests.

Re: The day I started believing in unit tests

#220

Post by randomdata »

MoreQARespect wrote: post #207
randomdata wrote: post #162

Earlier quoted context omitted.

> The idea that unit testing should be the default go to test I find to be horrifying. Kent Beck, who invented the term unit test, was quite clear that a unit test is a test that exists independent of other tests. In practice, this means that a unit test won't break other tests. I am not sure why you would want anything other than unit tests? Surely everyone agrees that one test being able to break another test is a…

>Kent Beck, who invented the term unit test, was quite clear that a unit test is a test that exists independent of other tests I vaguely remember him also complaining that there were too many conflicting definitions of unit tests. Maybe that can be solved with another definition? https://xkcd.com/927/ or maybe not. I dont know many people who would describe a test that uses playwright and hits a database as a unit te…

> He may have coined the term but that does not mean he owns it.

Certainly not, but there is no redefinition that is anything more than gobbledygook. Look at the very definition you gave: That's not a unique or different way to write tests. It's not even a testing pattern in concept. That's just programming in general. It is not, for example, unusual for you to use an alternative database implementation (e.g. an in-memory database) during development where it is a suitable technical solution to a technical problem, even outside of an automated test environment. To frame it as some special unique kind of test is nonsensical.

If we can find a useful definition, by all means, but otherwise what's the point? There is no reason to desperately try to save it with meaningless words just because it is catchy.

Post reply on HN