Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software testing, and why I'm unhappy about it

nhaehnle.blogspot.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Software testing, and why I'm unhappy about it

#31

Post by urthor »

invalidname wrote: post #19
ramesh31 wrote: post #18

Earlier quoted context omitted.

>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO. https://cucumber.io/docs/gherkin/reference/

This sounds like a good theory but the practice of it is really hard. Pretty quickly you end up with tests that "say" one thing but have nuanced different behavior in the underlying implementation. Then try to debug a "document"... I like the idea. But having tried it at scale, it becomes a mess. Code I can understand. I can read English comments. I can't debug English.

https://www.scalatest.org/at_a_glance/WordSpec

Seems pretty straightforward.

Re: Software testing, and why I'm unhappy about it

#32

Post by cranium »

drewcoo wrote: post #9

Doctor, it hurts when I punch myself in the head! If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them. Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff. T…

It would work for most "classical" software development. In this case, the author talks about conformance tests (a HUGE collection) from an external vendor. Most of them will fail at first, then you make them pass slowly but steadily.

The problem becomes: I want to know if there are significant regressions in the vendor tests, ie. tests that were green for a long time and suddenly changed. You could flag any test that became green at some point as "required" to pass the CI, but then you have tests that randomly succeed or fail depending on code you have not yet written (eg. locking around concurrent structures). Marking these tests manually is impractical and could definitively be replaced by tooling that supports some statistical modeling of success/failure.

You may have the best testing strategy for internal code but as long as you have to test against these conformance tests it's simply unfeasible to say "sorry, only green allowed".

Re: Software testing, and why I'm unhappy about it

#33

Post by invalidname »

urthor wrote: post #31
invalidname wrote: post #19

Earlier quoted context omitted.

This sounds like a good theory but the practice of it is really hard. Pretty quickly you end up with tests that "say" one thing but have nuanced different behavior in the underlying implementation. Then try to debug a "document"... I like the idea. But having tried it at scale, it becomes a mess. Code I can understand. I can read English comments. I can't debug English.

https://www.scalatest.org/at_a_glance/WordSpec Seems pretty straightforward.

Then why all the comments?

I know what typical code does. This code looks simple but that's misleading when you're trying to understand a failure. You want consistency and clarity. You want readablity like code is readable not like a book is readable.

Re: Software testing, and why I'm unhappy about it

#34

Post by gampleman »

Seems to me like you're underinvesting in tooling. It's a mistake a lot of development shops make - you focus on your product, so you can't spend time building something completely orthogonal, but in the process you suddenly waste man-years wasting time on a broken PR process, instead of spending a month early on building some tooling that would have removed the pain in the first place.

Re: Software testing, and why I'm unhappy about it

#35

Post by drothlis »

Gigachad wrote: post #4
drothlis wrote: post #2

Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).

Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.

Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.

Re: Software testing, and why I'm unhappy about it

#36

Post by kubanczyk »

drewcoo wrote: post #9

Doctor, it hurts when I punch myself in the head! If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them. Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff. T…

> take the human out of the "wait for green, then submit PR"

It'd be great if GitHub could open a PR for reviews (aka un-draft) automatically after CI succeeds. (If not in the core product, is there a bot that does that?)

Re: Software testing, and why I'm unhappy about it

#37

Post by ranting-moth »

> The above development practice works well when the SUT and TB are both defined by the same code repository and are developed together.

I once witnessed a team creating an app, specs and tests in three respective repositories. For no other reason than "each project should be in it's own repository".

The added work/maintenance around that is crazy, for absolutely no gain in that case.

Re: Software testing, and why I'm unhappy about it

#38

Post by midasz »

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

I always use (if the scenario is simple enough, which most are):

@Test

public void myTestMethod_Scenario_ShouldReturnThis() {....

Re: Software testing, and why I'm unhappy about it

#39

Post by arkh »

pjerem wrote: post #28
vincnetas wrote: post #26

Earlier quoted context omitted.

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable docu…

Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a littl…

> I’m sick of people escaping writing documentation

First level of documentation would be specifications. Which can then be used to write tests.

But in many, many shops "Agile" means "we don't do specs anymore, woohoo!".

Re: Software testing, and why I'm unhappy about it

#40

Post by sirsinsalot »

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention.

I don't think it quite does it right, but it is of note.

Post reply on HN