Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Questions to ask yourself when writing tests

charemza.name

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Questions to ask yourself when writing tests

#81

Post by KirinDave »

crdoconnor wrote: post #45
KirinDave wrote: post #17

Earlier quoted context omitted.

That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.

5x 5 minute manual test Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.

And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket?

How many customer minutes lost = 1 developer minute?

That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.

Re: Questions to ask yourself when writing tests

#82

Post by crdoconnor »

KirinDave wrote: post #81
crdoconnor wrote: post #45

Earlier quoted context omitted.

5x 5 minute manual test Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.

And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.

Yes, I would. However:

A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug.

B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level).

I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma).

>How many customer minutes lost = 1 developer minute

Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true.

I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute.

As I mentioned above, I really don't think it pays to be a test fundamentalist.

>That said, I think tests are the most expensive and most brittle way to address this problem.

Or a type system fundamentalist.

Re: Questions to ask yourself when writing tests

#83

Post by falsedan »

albertgoeswoof wrote: post #63
falsedan wrote: post #57

Earlier quoted context omitted.

Let me help expand your imagination: what if the code is for sshing to prod and needs a ssh-agent or a password typed in? Would you leave those cress on your CI system? Would Security be happy with that?

Why would a test build ever ssh to production? If you absolutely need to test connecting to an external system you should mock it or create an environment for testing against. Your credentials should also be stored as env variables so the test ones are used only for the test environment.

Because you're developing automated tools to deploy to prod & you're verifying the correctness of the system end-to-end with a no-op command. The connecting to an external system is not what's under test, and assume I've written unit tests that mock out the sshing, and that even with a staging environment I have to smoke test the new code before switching everyone over to the new version.

Where are those env variables stored on the CI system? In the job config? I would use Vault for this.

Re: Questions to ask yourself when writing tests

#84

Post by atticusCr »

viraptor wrote: post #52
atticusCr wrote: post #13

The author does not cover any question related to application security. Things like is this parameter/input value properly sanitized, does this piece is/is not vulnerable to injection attacks, does this piece of code performs authentication/authorization checks? Is RBAC properly implemented for this method?

I agree with some cases, but "is this parameter/input value properly sanitized" is a bit weird. It should only every apply to a) the db framework, b) those N really weird cases that have to break the abstraction and don't use the db framework. If you have to test every input, then the problem is on a completely different level than missing a test.

Kind of, if you have a centralized place to perform input data validation, as it should, then it is just a matter to test that piece of code same if you are using a framework. However, I don't understand why you refer to a db in the first place? Is it because I used the injection attack as an example? if that's the case bare in mind that Injection target other interprets as well not only a db.

But getting back to my original idea, what I want to highlight the need of adding cases to cover application security.

Re: Questions to ask yourself when writing tests

#85

Post by newfoundglory »

arkh wrote: post #47
Ace17 wrote: post #40

Earlier quoted context omitted.

> can you expand on why this is bad? Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. > Doesn't efficiency outweigh respecting feature boundaries in tests? In the end, what we're trying to minimize here is feature development time. This of course depends on the feedback loop duration (build + r…

> Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. Tests are not a debug tool. Tests are here to tell you when you broke something. When this happen you can get your debugging toolbox out: stack traces, profilers and things like GDB. And then follow the steps your test script did.

I disagree and am horrified by the dystopic image of our entire test suite having two possible results: "everything passed" or "something is wrong".

Re: Questions to ask yourself when writing tests

#86

Post by KirinDave »

crdoconnor wrote: post #82
KirinDave wrote: post #81

Earlier quoted context omitted.

And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.

Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (i…

Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests.

Miserably bad. Unforgivably bad.

Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing.

Because otherwise, all we'll do is continue to wallow in pride and failure, claiming it can't be helped. All the while using language like "case by case," that I have taken to mean: "I will never do that unless you force me to."

Fortunately, I think the scope of failure and fraud in the software industry has grown so late that folks are starting to take correctness as a requirement and not a nice to have. Another Equifax or two and maybe a nice DAO hack or something and folks are going to start saying, "Maybe it's just too bad we all learn to make bad software," turning to new techniques and practices.

Far-fetched? Maybe. But it is happening with AI...

Re: Questions to ask yourself when writing tests

#87

Post by crdoconnor »

KirinDave wrote: post #86
crdoconnor wrote: post #82

Earlier quoted context omitted.

Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (i…

Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests. Miserably bad. Unforgivably bad. Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing. Because otherwise, all we'll do is continue to wallo…

>that I have taken to mean: "I will never do that unless you force me to."

I actually created my own open source BDD framework and a ton of tooling to help automate stories.

19 out of 20 was a pretty conservative estimate of how much I automate - it's probably more like 39 out of 40. I'm a little obsessive about it because I want to dogfood my work properly so I automate quite a few things where the cost/benefit for a normal programmer would seem a bit low.

I'm very cognizant that the industry as a whole is terrible at testing and I'm hoping my software can one day do a small part to help with that.

Re: Questions to ask yourself when writing tests

#88

Post by arkh »

newfoundglory wrote: post #85
arkh wrote: post #47

Earlier quoted context omitted.

> Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. Tests are not a debug tool. Tests are here to tell you when you broke something. When this happen you can get your debugging toolbox out: stack traces, profilers and things like GDB. And then follow the steps your test script did.

I disagree and am horrified by the dystopic image of our entire test suite having two possible results: "everything passed" or "something is wrong".

Not two results.

But "when you click on this after that this happen while it shouldn't" is usually enough info to start debugging: you have reproducible steps. Which you can do with a debugger running so you see exactly were and how things break.

And it is a lot less brittle than "well we tried to refactor some minor thing and now everything is red; but we don't know if the software behavior changed or just because our tests were just checking the implementation".

Re: Questions to ask yourself when writing tests

#89

Post by viraptor »

atticusCr wrote: post #84
viraptor wrote: post #52

Earlier quoted context omitted.

I agree with some cases, but "is this parameter/input value properly sanitized" is a bit weird. It should only every apply to a) the db framework, b) those N really weird cases that have to break the abstraction and don't use the db framework. If you have to test every input, then the problem is on a completely different level than missing a test.

Kind of, if you have a centralized place to perform input data validation, as it should, then it is just a matter to test that piece of code same if you are using a framework. However, I don't understand why you refer to a db in the first place? Is it because I used the injection attack as an example? if that's the case bare in mind that Injection target other interprets as well not only a db. But getting back to my…

Yeah, my mind substituted parameter with query parameter. Too much database stuff at my $dayjob recently and I get tunnel vision ;-)

Re: Questions to ask yourself when writing tests

#90

Post by wst_ »

vog wrote: post #78
Ace17 wrote: post #39

Earlier quoted context omitted.

> ... micro level unit testing, driven by people like Uncle Bob. "The structure of your tests should not be a mirror of the structure of your code. The fact that you have a class named X should not automatically imply that you have a test class named XTest." "The structure of the tests must not reflect the structure of the production code, because that much coupling makes the system fragile and obstructs refactoring.…

I agree that these types of tests are very good and usefuly, but: Don't they use the wrong term for them? I believe these tests are more commonly called "integration tests", not "unit tests". Or, did I misunderstand something?

Integration tests are testing multiple modules and the way how they cooperate. These are mora a feature tests. Take a module as your unit, mock externals, give it an input and check the output/outcome.
Post reply on HN