Software testing, and why I'm unhappy about it • Hacker News Board

Re: Software testing, and why I'm unhappy about it

#11

Post by emptysea » Thu, Jan 19, 2023, 4:18 AM UTC

wpietri wrote: post #8

emptysea wrote: post #7

Earlier quoted context omitted.

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…

That sounds like it works, but shouldn't flaky tests be detected by frequent automatic test suite runs?

yeah I think the system is a combination, although I'm not sure the intricacies tbh

I've heard google does something fancier where they take a test, run it a bunch after it fails to check if it's failed

I think the system at work only runs each test a couple times before giving up and marking it failed

Re: Software testing, and why I'm unhappy about it

#12

Post by idontwantthis » Thu, Jan 19, 2023, 4:19 AM UTC

emptysea wrote: post #7

mixedCase wrote: post #3

> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based…

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…

That sounds fantastic. What tool are you describing?

Re: Software testing, and why I'm unhappy about it

#13

Post by Gigachad » Thu, Jan 19, 2023, 4:25 AM UTC

emptysea wrote: post #7

mixedCase wrote: post #3

> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based…

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…

Our setup just reruns the tests a few times which sorts out flaky tests. The page then shows the most frequently failing tests so they can be properly fixed.

Re: Software testing, and why I'm unhappy about it

#14

Post by andreareina » Thu, Jan 19, 2023, 4:33 AM UTC

The continuous testing is something I’ve thought about and it’s a tricky one. We use property tests[1] so here’s a quick stab at how I’d like it to look like:

Test starts failing, immediately send a report with the failing input, then continue with the test case minimisation and send another report when that finishes.

Concurrently, start up another long running process to look for other failures, skipping the input that caused the previous failure. We do want new inputs for the same failure though. This is the tricky one. We could probably make it work by having the prop test framework not reuse previously-failing inputs, but that’s one of the big strategies it uses to catch regressions.

[1] specifically, hypothesis on python

Re: Software testing, and why I'm unhappy about it

#15

Post by emptysea » Thu, Jan 19, 2023, 4:36 AM UTC

idontwantthis wrote: post #12

emptysea wrote: post #7

Earlier quoted context omitted.

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…

That sounds fantastic. What tool are you describing?

It's some home grown UI glued on top of Jenkins

This is pretty much the one feature that's nice, otherwise it's like a worse version of circle/github actions

Re: Software testing, and why I'm unhappy about it

#16

Post by jamesfinlayson » Thu, Jan 19, 2023, 4:41 AM UTC

emptysea wrote: post #7

mixedCase wrote: post #3

> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based…

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…

I think GitHub does something similar - public website tests must always pass but if you break GitHub Enterprise you get three days to fix it (or something like that - I think they had a blog post on it).

Re: Software testing, and why I'm unhappy about it

#17

Post by 13of40 » Thu, Jan 19, 2023, 5:32 AM UTC

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant:

You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result.

Because if you just throw in some code, you're only giving the poor bastard investigating it two puzzles to debug instead of one.

Re: Software testing, and why I'm unhappy about it

#18

Post by ramesh31 » Thu, Jan 19, 2023, 5:53 AM UTC

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result.

This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO.

https://cucumber.io/docs/gherkin/reference/

Re: Software testing, and why I'm unhappy about it

#19

Post by invalidname » Thu, Jan 19, 2023, 6:19 AM UTC

ramesh31 wrote: post #18

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO. https://cucumber.io/docs/gherkin/reference/

This sounds like a good theory but the practice of it is really hard. Pretty quickly you end up with tests that "say" one thing but have nuanced different behavior in the underlying implementation.

Then try to debug a "document"...

I like the idea. But having tried it at scale, it becomes a mess. Code I can understand. I can read English comments. I can't debug English.

Re: Software testing, and why I'm unhappy about it

#20

Post by randomdata » Thu, Jan 19, 2023, 6:20 AM UTC

13of40 wrote: post #17

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in s…

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation.

If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentation in an executable form (to be able to automatically validate that the documentation is true).