Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software testing, and why I'm unhappy about it

nhaehnle.blogspot.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Software testing, and why I'm unhappy about it

#41

Post by sirsinsalot »

randomdata wrote: post #23
ramesh31 wrote: post #18

Earlier quoted context omitted.

>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO. https://cucumber.io/docs/gherkin/reference/

Gherkin/Cucumber reintroduce the very problem TDD/BDD was intended to solve: Documentation falling out of sync with the implementation. The revelation of TDD, which was later rebranded as BDD to deal with the confusion that arose with other types of testing, was that if your documentation was also executable the machine could be used to prove that the documentation is true. The Gherkin/Cucumber themselves are not exe…

TDD wasn't rebranded BDD.

BDD is a QA concern, primarily used for QA tests against a written (BDD) requirement.

TDD is about unit testing, which is about testing the implementation BY developers FOR other developers.

TDD says nothing about the correctness of the software against a spec, only that a given implementation aligns with a developer's intention.

Re: Software testing, and why I'm unhappy about it

#42

Post by rightbyte »

pjerem wrote: post #28
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a littl…

Not writing docs is a sort of hazing. Also job security.

Re: Software testing, and why I'm unhappy about it

#43

Post by saiya-jin »

pjerem wrote: post #28
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a littl…

> But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming.

I completely agree with you, ie right now doing bunch of data migration code that is awful 200 lines on first look, but does quite clever transformations, handles various data corner cases, manages lots of threads, is already quite optimized (had 30x speed increase just over last week's state and not yet done with it) etc. and... is full of little green one-liners explaining why certain logic is happening, why at given place, and not elsewhere, and how it helps later in the code.

Its even one-off migration, and its mostly for me only. But I still put comments in, have enough experience to know I will keep using those comments in further optimizations, and I know by heart that many one-off efforts end up being re-used later. Code dense with logic shouldn't require you to re-read it all to have constant full mental model of it and all its branching and possibilities just because you want to tweak it a bit.

The important point is to evolve those comments with code, otherwise they become worse than no comments at all. This is where most folks hit the wall - they are simply too lazy or undisciplined for that.

Re: Software testing, and why I'm unhappy about it

#44

Post by BerislavLopac »

sirsinsalot wrote: post #41
randomdata wrote: post #23

Earlier quoted context omitted.

Gherkin/Cucumber reintroduce the very problem TDD/BDD was intended to solve: Documentation falling out of sync with the implementation. The revelation of TDD, which was later rebranded as BDD to deal with the confusion that arose with other types of testing, was that if your documentation was also executable the machine could be used to prove that the documentation is true. The Gherkin/Cucumber themselves are not exe…

TDD wasn't rebranded BDD. BDD is a QA concern, primarily used for QA tests against a written (BDD) requirement. TDD is about unit testing, which is about testing the implementation BY developers FOR other developers. TDD says nothing about the correctness of the software against a spec, only that a given implementation aligns with a developer's intention.

If unit tests are not testing the behaviour, it's being done wrong.

If they are, the only difference between TDD and BDD is where, in which form, and by whom is that behaviour defined.

Re: Software testing, and why I'm unhappy about it

#45

Post by sirsinsalot »

BerislavLopac wrote: post #44
sirsinsalot wrote: post #41

Earlier quoted context omitted.

TDD wasn't rebranded BDD. BDD is a QA concern, primarily used for QA tests against a written (BDD) requirement. TDD is about unit testing, which is about testing the implementation BY developers FOR other developers. TDD says nothing about the correctness of the software against a spec, only that a given implementation aligns with a developer's intention.

If unit tests are not testing the behaviour, it's being done wrong. If they are, the only difference between TDD and BDD is where, in which form, and by whom is that behaviour defined.

Unit tests assert implementation behaviour to aid refactoring. If developers misunderstand the spec, the unit tests can be valid. They don't assert developer understanding.

Say it with me, unit tests are to aid refactoring.

If we mix QA and implementation details just because both sides use the word "test" it ends in trouble.

QA should be blind to unit test coverage or even usage at all, they're totally independent concerns.

A passing unit test says nothing against correctness of product against a spec or design... only that it works and continues to work as a developer intended, to aid the work of future developers, even if they misunderstood the spec.

Your comment is at the core of why QA is a total mess. Everyone is confused about what "test" means in different contexts.

Why have a QA function at all with 100% unit test coverage? Because the unit tests may encode misunderstanding by developers. They're there to fight entropy, not wrongness.

QA, using BDD and other tools, ensure the product is correct, regardless of how well it fights entropy by unit tests.

Re: Software testing, and why I'm unhappy about it

#46

Post by nurettin »

If I am given the time and resources I do this:

Phase 1. Code and test basic functions concerning any kind of arithmetic, mathematical distribution, state machines, file operations and datetimes. This documents any assumptions and makes a solid foundation.

Phase 2. Write a simulation for generating randomized inputs to test the whole system. Run it for hours. If I can't generate the inputs, find as big a variety of inputs as possible. Collect any bugs, fix, repeat. This reduces the chances of finding real time bugs by three orders of magnitude.

This has worked really well in the past whether I'm working on games, parsers or financial software. I don't conform to corporate whatever driven testing patterns because they are usually missing the crucial part 2 and time part 1 incorrectly.

Re: Software testing, and why I'm unhappy about it

#47

Post by Gigachad »

drothlis wrote: post #35
Gigachad wrote: post #4

Earlier quoted context omitted.

Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.

Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.

In that case you have to release versions with compatibility for both the new and old way. At no point can I ever see it being a good idea to just let tests fail.

Re: Software testing, and why I'm unhappy about it

#48

Post by amelius »

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

I suppose soon you could ask GPT "what is this code supposed to do?"

(I would buy a Copilot subscription for this)

Re: Software testing, and why I'm unhappy about it

#49

Post by yellow_lead »

theamk wrote: post #10

The author's problem is pretty simple: the test repo is required for pre-merge tests to pass, but it can be updated independently, without having pre-merge tests pass. And the answer is pretty simple: pin the specific test repo version! Use lockfiles, or git submodules, or put "cd tests && git checkout 3e524575cc61" in your CI config file _and keep it in the same repo as source code_ (that part is very important!). T…

Yeah - developers need to control their own tests. If in the weird case they don't control their tests (conformance tests) - you need to control when those tests are added.

Re: Software testing, and why I'm unhappy about it

#50

Post by nosianu »

amelius wrote: post #48
13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

I suppose soon you could ask GPT "what is this code supposed to do?" (I would buy a Copilot subscription for this)

Which would replace all those humans producing perfectly valid sounding explanations that if you invest some research effort have no basis in (the usually far more complex, but also far more fascinating and infinitely deep) reality. So yes, I think AI can indeed replace lots of human-produced thoughts :-)

I admit to have been guilty of this myself. I have a famous anecdote-example where I had a very well-paid contractor job and explained something about how my then department's software worked to someone from another department. I think I must have sounded very convincing, the person went off to change something in how they used our stuff. A few minutes later, after accidentally meeting and casually chatting with my boss for that job I realized everything I had said was total garbage. I quickly excused myself from my boss and hurried after the person to tell them to forget and ignore everything I had just explained to them because it was all wrong. I think this last step is not what happens in those cases because we don't usually realize that such a thing just happened.

The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link?

Post reply on HN