Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The day I started believing in unit tests

mental-reverb.com

Post reply on HN
Print view View original

251–260 of 269 posts

Re: The day I started believing in unit tests

#251

Post by lizardking »

agentultra wrote: post #39

It's all degrees. Unit tests are great at finding examples of errors or correct behaviours. However they prove nothing and they definitely do not demonstrate the absence of errors. They are often sufficient for a great deal of projects. If all it takes to convince you it's "good enough," are a handful of examples then that's it. As much as you need and no less. However I find we programmers tend to be a dogmatic bunc…

They can prove that by building out some new feature you implicitly break some existing functionality that didn't occur to the person writing the requirements.

Re: The day I started believing in unit tests

#252

Post by randomdata »

MoreQARespect wrote: post #242
randomdata wrote: post #220

Earlier quoted context omitted.

> He may have coined the term but that does not mean he owns it. Certainly not, but there is no redefinition that is anything more than gobbledygook. Look at the very definition you gave: That's not a unique or different way to write tests. It's not even a testing pattern in concept. That's just programming in general. It is not, for example, unusual for you to use an alternative database implementation (e.g. an in-m…

The definition I gave is the one people use. Hate or love it youre not going to change it to encompass end to end tests and neither will Kent Beck. It's too embedded.

> youre not going to change it

I might. I once called attention to the once prevailing definition of "microservices" also not saying anything. At the time I was treated like I had two heads, but sure enough now I see a sizeable portion (not all, yet...) of developers using the updated definition I suggested that actually communicates something. Word gets around.

Granted, in that case there was a better definition for people to latch onto. In this case, I see no use for the term 'unit test' at all. Practically speaking, all tests people write today are unit tests. 'Unit' adds no additional information that isn't already implied in 'test' alone and I cannot find anything within the realm of testing that needs additional differentiation not already captured by another term.

If nothing changes, so what? I couldn't care less about what someone else thinks. Calling attention to people parroting terms that are meaningless is entirely for my own amusement, not some bizarre effort to try and change someone else. That would be plain weird.

Re: The day I started believing in unit tests

#253

Post by pfdietz »

To me an interesting distinction is not between unit and integration tests, but between tests that are run quickly as part of a gate on commits in CI, vs. tests that are run more asynchronously searching for bugs.

The former must run quickly, and it's ok if the exact same test is run over and over. The latter need not run quickly, but benefits if new tests can be created and run, or if the tests incorporate randomness so they don't do the same thing each time they are run.

Here, it seems he was using tests intended for the first purpose for the second purpose instead. That can work, as it did here, but I don't think it's optimal. Better to have more exploratory, randomized, property-based tests chugging away in the background to find weird new ways the code can fail.

Re: The day I started believing in unit tests

#254

Post by benrutter »

gorgoiler wrote: post #215
feoren wrote: post #130

Earlier quoted context omitted.

But you literally cannot possibly test that assertion for all x . Let's take a slightly harder problem: prove (or at least test conclusively) that for all integer x , the output y of the following function is always even: y = x^2 + x + 2 There is essentially no way to prove this for all x by simply testing all integers. If your integers are 64-bit, you don't have enough time in the lifespan of the universe. On the ot…

Could you do this in Python (using positive integers as an example of a type rather than even numbers)?: class N(int): def __init__(self, z: int): assert z > 0 super().__init__(z) def log2(n: N) -> float: … log2(N(32)) # 5.0 log2(32) # type error log2(N(-32)) # runtime error You are still relying on the runtime to detect errors and it’s annoying to have to cast all ints to Ns, but you at least won’t ever take the log…

That's a cool trick! Depending on your use case though it might not go far enough since the new "type" is really just an integer with an assert. It won't be picked up by static type checkers and there's nothing stopping you doing this:

``` x = N(2) x -= 10 ```

I think x still winds up being less than 0 here?

Its probably easier just to add an "assert" with a friendly message at the start of the log function.

Re: The day I started believing in unit tests

#255

Post by pfdietz »

throwaway2037 wrote: post #234
pfdietz wrote: post #123

Earlier quoted context omitted.

Why isn't the question "does adequate testing dramatically reduce the benefits of static typing" asked? Why is static typing privileged by default?

"[A]dequate testing": Woah, I love this term. It is judgmental right from the start. It's right up there with "convention over configuration" and "well, if you wear your face mask _correctly_..." I once saw a blog post from an embedded programmer talking about how difficult it is to write "adequate" unit tests for embedded code. If you are writing code that will run in a heart pace maker or aeroplane auto-pilot/lande…

It's not judgmental in some dubious way. "Adequate" here means adequate to ensure the software achieves a specified (and high) level of reliability.

If testing is enough to ensure the software is reliable, does the extra benefit of static typing make it worth the cost? This could be quite a lot of testing! The more testing that is done, the fewer type bugs remain that static typing would have found.

The argument you want to make against this is that static typing would have benefit beyond just finding bugs. The argument you don't want to make is that static typing reduces the need for testing.

Re: The day I started believing in unit tests

#256

Post by pfdietz »

tshaddox wrote: post #189
pfdietz wrote: post #123

Earlier quoted context omitted.

Why isn't the question "does adequate testing dramatically reduce the benefits of static typing" asked? Why is static typing privileged by default?

Probably because using a static type system gives you those benefits "for free," whereas unit tests are things you need to write and maintain. ("For free" in scare quotes because of course there are always tradeoffs between different programming languages.)

The counterargument is that for sufficiently reliable software extensive testing is needed anyway, and if you do that you find the type errors "for free".

If the testing is insufficient, as in practice it often surely is, then static typing would seem to be more valuable.

Re: The day I started believing in unit tests

#257

Post by kcartlidge »

berkes wrote: post #43
atticora wrote: post #38

Earlier quoted context omitted.

My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. Yes, they make testing a lot slower. Our main app takes around 40 minutes to build which isn't good. I'd like it to be faster. But writing a bunch of separate integration tests to cover those functions would…

> My "unit tests" do hit the database and file system, and I have found and fixed many many problems during testing by doing so. I have found many other problems with those calls in production when I didn't do so. No-one said that integration tests can't also be very valuable. From the little context I get that you write integration tests, and that is fine. They are useful, valuable! But they are not unit-tests. edit…

I find the easiest way (for me) to identify what type of test is running is by looking at responsibility.

- A unit test is single responsibility. It tests just that one bit of code with all dependencies stubbed, abstracted, mocked, or removed from consideration in some way.

- An integration test is multiple responsibility. It tests just one bit of functionality as a vertical slice through the stack (including [only] relevant dependencies) with all other aspects of the code base eliminated from consideration.

- An end to end test is full responsibility. It tests a complete path through all the functionality necessary to complete a 'journey' as a user/consumer of the app/tool.

So for example VAT calculation is unit tested as isolated code, invoicing is integration tested as a vertical slice including database etc, and order processing is end-to-end tested from placing the order through to its completion.

That's a simplified example and not always accurate depending upon the system and the team perspectives/opinions, but the principle of looking at responsibilities is a very useful rule of thumb.

Re: The day I started believing in unit tests

#258

Post by HankB99 »

I like them because they help me to partition my code into units that are easier to write and test. Once they're working, assembling the parts generally leads to a working project.

It also motivates me to get small pieces working and tested before I get to the finish line. Each successful test is a victory!

Re: The day I started believing in unit tests

#259

Post by bobthepanda »

Verdex wrote: post #52
bedobi wrote: post #31

Earlier quoted context omitted.

> confirm that the implementation is consistent between small changes without weird side effects not sure what this is referring to, but I'll give an example say you have a requirement that says if you call POST /user with a non-existing user, a user should be created and you should get a 2xx response with some basic details back you could test this by actually hitting the endpoint with randomly generated user data k…

To be fair to unit tests, I really like them for making sure that complicated code gets tested thoroughly. However, very often the complicated code isn't isolated to a single unit. It instead lives distributed amongst multiple objects that are reused for several competing aspects and all have their own undocumented assumptions about how the world works. Now maybe this implies that we need a wide scale change in codin…

Unit and integration testing are not strategies to be used exclusively.

Integration tests relying on interlocking behavior are, by their nature, complicated. Unit tests are there to test what can be tested simply, and are cheaper to write, so your test structure should be pyramid shaped, with hopefully fewer tests as complexity increases.

Re: The day I started believing in unit tests

#260

Post by RaftPeople »

randomdata wrote: post #228
RaftPeople wrote: post #226

Earlier quoted context omitted.

Every step updates shared databases (frequently plural). In the case of the fulfillment step, the following systems+databases were involved: ERP, WMS, Shipping. Typically, in end to end testing, tests are run within the same shared QA system and are semi-isolated based on choice of specific data (e.g. customers, products, orders, vendors, etc.). If this test causes a different test to fail, or vice-versa, then you ha…

> Every step updates shared databases (frequently plural). That's fine. It all happens within a single unit. A unit should mutate shared state within the unit. Testing would be pretty much useless without. > If we call that entire sequence of steps a "unit" test, would you start with testing the entire sequence of steps, or would you recommend testing the individual steps first? For all intents and purposes, you can'…

> For all intents and purposes, you can't test the individual steps.

Sure you can, and we did (that is a real example of an end to end test from a recent project) which also included testing the individual steps in isolation, which was preceded by testing the individual sub-steps/components of each step (which is the portion that is typically considered unit testing).

For example, step 1 is broken down into the following sub-steps which are all tested in isolation before testing the combined group together:

1.1-Calculate the current on hand inventory from all locations for all products

1.2-Calculate the current in transit inventory for all locations for all products

1.3-Calculate the current open inventory reservations by business partner and products

1.4-Calculate the current in process fulfillments by business partner and product

1.5-Resolve the configurable inventory feed rules for each business partner and product (or product group)

1.6-Using the data in 1.1 through 1.5, resolve the final available qty for each business partner and product

1.7-Construct system specific messages for each system and/or business partner (in some cases it's a one to one between business partner and system, but in other cases one system manages many business partners).

1.7.1-Send to system B

1.7.2-Send to system C

1.7.3-Send to system D

1.7.N-etc.

> And the product of step one is undoubtedly internal state, so there is no way for the test to observe the state change in isolation

The result of step 1 is that over in software system B (an entirely different application from system A) the inventory availability for each product from system A is properly represented in the system. Meaning queries, inquiries, reports, application functions (e.g. Inventory Availability by Partner), etc. all present the proper quantities.

To validate this step, it can be handled one of two ways:

1-Some sort of automated query that extracts data from system B and compares to the intended state from step 1 (probably by saving that data at the end of that step).

or 2-A user manually logs in to system B and compares to the expected values from step 1 (again saved or exposed in some way). This method works when the number of products is purposefully kept to a small number for testing purposes.

> If the individual steps can be tested individually (ignoring a case of you doing something stupid), it's not actually and end-to-end process, so your example would make no sense. Granted, we have already questioned if it is a bad example.

Yes the individual test can be tested in individually. Yes it is an end to end test.

> Granted, we have already questioned if it is a bad example.

It's a real example from a real project and it aligns with the general notion of an end to end test used in the industry.

More importantly, combined with the unit tests, functional tests, integration tests, performance tests, other end to end tests and finally user acceptance tests, it contributed to a successful go-live with very few bugs or design issues.

Post reply on HN