Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software testing, and why I'm unhappy about it

nhaehnle.blogspot.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Software testing, and why I'm unhappy about it

#51

Post by mike_hearn »

kubanczyk wrote: post #36
drewcoo wrote: post #9

Doctor, it hurts when I punch myself in the head! If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them. Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff. T…

> take the human out of the "wait for green, then submit PR" It'd be great if GitHub could open a PR for reviews (aka un-draft) automatically after CI succeeds. (If not in the core product, is there a bot that does that?)

My company uses a workflow where we don't use PRs for code reviews. Instead we each have our own git repo that's a fork of the tech lead's, with some git rules in place to impose a branch namespace. To open a review request you push a branch into the reviewer's repository. Our CI system detects the new branch and starts running it. Once CI passes that updates the bug tracker which triggers a notification to the reviewer.

The reviewer then does a git fetch, and then checks out the newly created rr/ branch. They make any small changes that aren't worth a roundtrip and push them to the rr branch. They add FIXME comments for bigger changes. They then either assign the ticket back to the developer, or go ahead and merge straight into their own dev branch. Once an rr branch is merged it's simply deleted. The dev branch is then pushed and CI will merge it to that user's master when it's green.

IntelliJ will show branches in each origin organized by "folder" if you use backslashes in branch names, and gitolite (which is what we use to run our repos) can impose ACLs by branch name too. So for example only user alice can push to a branch named rr/alice/whatever in each persons repo. That ensures it's always clear where a PR/RR is coming from.

Because each user gets their own git repo and cloned set of individual CI builds, you can push experimental or WIP branches to your personal area and iterate there without bothering other people.

This workflow gets rid of things like draft PRs (which are a contradiction), it ensures each reviewer has a personal review queue, it means work and progress is tracked via the bug tracker (which understands commands in commit messages so you can mark bugs as fixed when they clear CI automatically) and it eliminates the practice of requesting dozens of tiny changes that'd be faster for the reviewer to apply themselves, because reviewer and task owner can trade commits on the rr branch using git's features to keep it all organized and mergeable.

Re: Software testing, and why I'm unhappy about it

#52

Post by ramesh31 »

sirsinsalot wrote: post #45
BerislavLopac wrote: post #44

Earlier quoted context omitted.

If unit tests are not testing the behaviour, it's being done wrong. If they are, the only difference between TDD and BDD is where, in which form, and by whom is that behaviour defined.

Unit tests assert implementation behaviour to aid refactoring. If developers misunderstand the spec, the unit tests can be valid. They don't assert developer understanding. Say it with me, unit tests are to aid refactoring. If we mix QA and implementation details just because both sides use the word "test" it ends in trouble. QA should be blind to unit test coverage or even usage at all, they're totally independent c…

>Your comment is at the core of why QA is a total mess. Everyone is confused about what "test" means in different contexts.

Particularly the important differences between unit, integration, and e2e testing. Many people use the words interchangeably when they are completely different concerns.

Unit test your library code that has no external dependencies, integration test your classes that implement those libraries, and e2e test your application that is built with those classes. There are varying philosophies to which are more useful, but it's an important distinction to maintain in terminology.

Re: Software testing, and why I'm unhappy about it

#53

Post by Phemist »

nosianu wrote: post #50
amelius wrote: post #48

Earlier quoted context omitted.

I suppose soon you could ask GPT "what is this code supposed to do?" (I would buy a Copilot subscription for this)

Which would replace all those humans producing perfectly valid sounding explanations that if you invest some research effort have no basis in (the usually far more complex, but also far more fascinating and infinitely deep) reality. So yes, I think AI can indeed replace lots of human-produced thoughts :-) I admit to have been guilty of this myself. I have a famous anecdote-example where I had a very well-paid contrac…

>The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link?

Relevant are Sperry & Gazzaniga's split brain experiments. Participants of these experiments had had their corpus callosum (one of the major "information" pathways between our brain's two halves) cut. This was an operation performed to keep epileptic seizures in check.

https://en.wikipedia.org/wiki/Split-brain

In these participants, specific brain "functions" such as speech were highly lateralized, meaning only one half of the brain was able to perform it to a satisfying degree.

Note that these were already not neuro-typical people prior to the experiments (given the regular, debilitating epileptic seizures), so reaching general conclusions from these experiments is hard.

Remember also that, like our brains, our bodies are also highly lateralized, such that the right-half of our brain controls the left-side of our body, and the left-half of the brain controls the right-side of our body. If you ever wanted proof against intelligent design, the way our brain connects to our eyes & body is one very strong argument..

Anyway, one experiment stands to mind where one half of the brain was instructed to perform some action (move the left arm, or something similar). Then the other half would be asked _why_ that arm was just moved. It would confabulate, on the spot, totally legit, but obviously bullshit, sounding reasoning. E.g. "I felt cold so I wanted to put on a coat", rather than "the experimenter instructed me to move it".

So, rather than claiming "I don't know", it would just make up a plausible reasoning. It is really unimaginable to _not_ know why you moved your arm..

Re: Software testing, and why I'm unhappy about it

#54

Post by alpaca128 »

vincnetas wrote: post #26
randomdata wrote: post #20

Earlier quoted context omitted.

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentatio…

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking.

Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice that's the kind of doc I tend to have the best experience with. For example the Rust stdlib docs are auto-generated but the language also supports notes and (automatically unit-tested) examples in docstrings which means the API docs are filled with explanations & examples and mentions what assumptions are made about inputs.

Re: Software testing, and why I'm unhappy about it

#55

Post by jeroenhd »

Gigachad wrote: post #4
drothlis wrote: post #2

Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).

Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.

I can't think of any project I've worked on where external test suites even make sense. I suppose it would work when you have a very clear spec or compliance document you could write independent tests, or if you're rewriting a system and need the public API to be bug-for-bug compatible with the old one, but other than those niche use cases I wouldn't want to keep those tests external at all.

Even if you do have external tests, you still need internal ones for the surface area your external tests don't check for. Unit tests and such don't make sense at all combined with a separate test repo.

Re: Software testing, and why I'm unhappy about it

#56

Post by hitchstory »

alpaca128 wrote: post #54
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice th…

I built this framework coz while I didn't believe in "code as documentation" I did believe that example based specifications, tests and documentation were all sort of the same thing (triality):

https://hitchdev.com/hitchstory/

The difference between this and behave/cucumber is that the A) specification language allows for more complex representations and B) there's a templating step to generate readable documentation.

Re: Software testing, and why I'm unhappy about it

#57

Post by kasey_junk »

pjerem wrote: post #28
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a littl…

The assumption in this that doesn’t fit my experience is that it assumes that someone that writes unreadable code abstracted over dozens of files, is going to be able to write clear, expressive and complete documentation.

In my experience if they can do the latter the former isn’t a problem. But since many people can’t you are left with bad code, littered with bad (often contradictory) comments which makes the problem worse not better.

Re: Software testing, and why I'm unhappy about it

#58

Post by magicalhippo »

vincnetas wrote: post #26
randomdata wrote: post #20

Earlier quoted context omitted.

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentatio…

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Code is documentation, but it only tells you a part of the story. Good comments can explain why, but without writing comment essays it's usually not sufficient.

And, as you note, when integrating systems you need more than just the code and comments, since the code might not even be written with the other system in mind.

Re: Software testing, and why I'm unhappy about it

#59

Post by hgomersall »

sirsinsalot wrote: post #40
13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention. I don't think it quite does it right, but it is of note.

We have a policy of making each test a spec. That is, a test requires a plain text spec to be attached to it in its doc string. It's kind of like BDD but without all the weird DSLs.

Re: Software testing, and why I'm unhappy about it

#60

Post by BlackFly »

sirsinsalot wrote: post #45
BerislavLopac wrote: post #44

Earlier quoted context omitted.

If unit tests are not testing the behaviour, it's being done wrong. If they are, the only difference between TDD and BDD is where, in which form, and by whom is that behaviour defined.

Unit tests assert implementation behaviour to aid refactoring. If developers misunderstand the spec, the unit tests can be valid. They don't assert developer understanding. Say it with me, unit tests are to aid refactoring. If we mix QA and implementation details just because both sides use the word "test" it ends in trouble. QA should be blind to unit test coverage or even usage at all, they're totally independent c…

Unit tests (class or method as the unit) hinder refactoring by binding to low level implementation details. When you refactor, by definition you are changing what the factors (units) are. Generally, your unit tests will then be testing implementation details that no longer exist. By strongly coypling to implementation details, unit test suites suffer an extremely large ripple effect on refactoring.

Tests in general can only help you refactor code at a lower level of granularity than what you are testing. Something lower than unit level is a contradiction.

Of course, you can instead test business behavior which isn't as volatile in refactoring and change your definition of unit to be a unit of practical business requirements...

Post reply on HN