Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test Failures Should Be Actionable

testing.googleblog.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Test Failures Should Be Actionable

#11

Post by gavmor »

neallindsay wrote: post #2

This is the main reason we have assertion libraries and don't just use boolean expressions. My favorite test framework of all time, rspec-given[1] (by the late, great Jim Weirich) solved this a different way by using AST introspection to extract both sides of a boolean expression and give a helpful message even with an assertion like: `Then { result == 7 }` [1]: https://github.com/jimweirich/rspec-given

Thank you! It feels a bit crazy-making to see a post like this published with no mention of eg rspec "matchers", ubiquitous in Ruby land for decades. I'm somewhat dismayed that this even needs to be said in 2024; I'll bet there's an entry in Ward Cunningham's wiki imploring devs to write tests that fail descriptively but—as evidenced by my lack of citations—it's often easier to write a post than search for a post. (And that's part of Joe Armstrong's "The Mess We're In")

Edit: to make a non-meta point, these sorts of appeals to "please exhibit good behavior" can't stand against economic forces, eg time constraints and KPIs, so I wonder if there isn't a ToolsNotRules[0] implementation which might enforce more introspective assertions.

Edit2: Oh, duh—TDD is the practice which enforces more actionable test failures! After all, an unactionable test failure drives no development. So if you're really keen on actionable failures, try TDD.

0. https://benchristel.github.io/process-to-processes/Fundament...

Re: Test Failures Should Be Actionable

#13

Post by burnte »

simiones wrote: post #10
burnte wrote: post #5

With the rapidly increasing number of websites and applications that feel "an error occurred" is an acceptable error message, I'm glad someone is pointing out that failures should contain actionable information, but I'm also greatly saddened this is needed. I truly do not understand how anyone, much less developers, needs to be told that the content of an error message is important. It really boggles my mind. How is…

For a website, if the cause of the error is something internal to the server (a bug or other temporary condition), what could possibly be gained by telling the user the reason why this failed? "An error occured, try again later" is the only actionable information you can possibly give them, if they have no access on your server to fix your bug.

How do I know if it's an internal error or if I did something wrong? And yes, showing users the internal errors is useful as they can report the issue with the error message, saving developers time in searching.

Re: Test Failures Should Be Actionable

#14

Post by burnte »

tichiian wrote: post #7
burnte wrote: post #5

With the rapidly increasing number of websites and applications that feel "an error occurred" is an acceptable error message, I'm glad someone is pointing out that failures should contain actionable information, but I'm also greatly saddened this is needed. I truly do not understand how anyone, much less developers, needs to be told that the content of an error message is important. It really boggles my mind. How is…

There are reasons for not reporting detailed errors to the user. For websites, security is a big one. Detailed error messages can lead to information exfiltration, exploits, fingerprinting and generally bad things. Long error messages in some network protocols enable amplification attacks. For general applications, there are commercial reasons. You don't want to enable your users too much, they should be incentivized…

While this is true, I don't think you truly missed my point. This is just being pedantic for the sake of it.

Re: Test Failures Should Be Actionable

#15

Post by clintonb »

burnte wrote: post #13
simiones wrote: post #10

Earlier quoted context omitted.

For a website, if the cause of the error is something internal to the server (a bug or other temporary condition), what could possibly be gained by telling the user the reason why this failed? "An error occured, try again later" is the only actionable information you can possibly give them, if they have no access on your server to fix your bug.

How do I know if it's an internal error or if I did something wrong? And yes, showing users the internal errors is useful as they can report the issue with the error message, saving developers time in searching.

Input validation is a separate thing. If a user provides bad input, yes, the server should return an appropriate error code and human-friendly message. _Lacking_ sufficient validation is a problem that could result in an internal server error that could be resolved by changing inputs, but the validation doesn’t exist so the server cannot inform you of this.

Re: Test Failures Should Be Actionable

#16

Post by lolc »

I'm often left scratching my head about what behaviour a test was even supposed to test! One thing I find very helpful is to add my expectation as string message to every assertion in the test. When the test fails, it will tell me which expectation was violated.

Re: Test Failures Should Be Actionable

#17

Post by tichiian »

burnte wrote: post #14
tichiian wrote: post #7

Earlier quoted context omitted.

There are reasons for not reporting detailed errors to the user. For websites, security is a big one. Detailed error messages can lead to information exfiltration, exploits, fingerprinting and generally bad things. Long error messages in some network protocols enable amplification attacks. For general applications, there are commercial reasons. You don't want to enable your users too much, they should be incentivized…

While this is true, I don't think you truly missed my point. This is just being pedantic for the sake of it.

You are right in that the world would be a better place with more useful and actionable error messages.

But:

> How is it not obvious to the point of pain that you need to know WHY something failed, not only that it failed?

This is obvious to everyone, at least after encountering the first few errors. But that makes it the perfect thing to upsell, to milk your existing customers with a pricier plan. One instance is e.g. Microsoft selling access to their customers' logs to those same customers: https://www.theregister.com/2023/07/20/under_cisa_spressures... (they backpedalled after it blew up spectacularly).

No, I don't like it. Yes, I hate those Ferengi. Changing how things are for the better not only requires recognizing the immediate problems but also why the problem is persisted, and who is to blame for it.

Post reply on HN