Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TDD Doesn't Work

blog.cleancoder.com

Post reply on HN
Print view View original

111–120 of 133 posts

Re: TDD Doesn't Work

#111

Post by bitL »

lisivka wrote: post #107
bitL wrote: post #102

Earlier quoted context omitted.

The thing is that there are problems we simply can't solve in theory nor in practice, yet we use approximate solutions all the time - and that is the case of advanced distributed algorithms. In theory, we simply can't handle real-world asynchronous systems. And when we pretend we have partially synchronous systems and build abstractions around them, they aren't 100% working. Now add in some complex bugs (like getting…

Each test case increases confidence in the system by small amount. Confidence never can approach 100% (because we need to predict future to achieve that), so no amount of testing can give you 100% confidence, only 80%, 95% (2x price), 98% (4x), 99,5% (8x), 99,95% (16x), 99,995% (32x), and so on. It's your message, right?

My message is more like your confidence after writing hundreds of thousands tests might be just 50%. From my own experience, every single bad case that can happen in a complex system will happen at some point at some customer, wrecking their system and costing them potentially millions, in serious trading bugs even leading to a bankruptcy. Your testing suite won't catch these initially but reactively when you add that test case to your regression suite after bad things happened. In complex systems, tests are just a heuristics for quality, not really something you can rely on (but it's way way better to have them than not). Often tests are clustered around low-hanging fruit or around parts of system used by developers or initial customers and any deviation in usage patterns can cause an outbreak of new, unexpected incorrect situations. Similarly, proving correctness using some formal verification tools might increase your confidence, but won't give you 100% either, as we simply can't model reality properly even within our own frameworks :(

Re: TDD Doesn't Work

#112

Post by unclebobmartin »

gozur88 wrote: post #34
liquidise wrote: post #2

Commenting on TDD stories here is historically a bad practice, but i'll add my input here. I have never let my teams go full TDD. The reason is that in all my experience, TDD sacrifices a lot of velocity for the sake of automated tests. When i hear about the reduction in total bugs injected, it is a "duh" moment. The fastest way to make a team inject 30% fewer bugs is to have them write 30% less code. That isn't snar…

That's been my experience. I would add that TDD is the antithesis of "agile", since any changes you make to your product will require changes to the tests. Sometimes large changes.

Only if you design your tests that way. Tests are just software. If a change to one part of a software system requires massive changes to another part of that same system; then the system is poorly designed. Indeed, that may be the very definition of poor design.

So if a change to your production code causes large changes to your test code, then one, or the other, or both are poorly designed. You have neglected the design. You have allowed couplings to proliferate.

Re: TDD Doesn't Work

#113

Post by unclebobmartin »

c3RlcGhlbnI_ wrote: post #32

Oh man, it is really funny that he ends it with telling people to read the study. To clarify the linked study is attempting to replicate https://dl.acm.org/citation.cfm?id=1070834 , THE seminal study in Test Driven Development. Well to be more precise it was replicating an existing replication of that study which failed to replicate the original results. They were trying to modify the design so as to account for issu…

I use that style, from time to time, because I like it. ;-)

Re: TDD Doesn't Work

#114

Post by westurner »

lisivka wrote: post #74
nercury wrote: post #51

Earlier quoted context omitted.

I was commenting on the methodology as I heard and watched it explained by the author (Robert C Martin), as well as the way it was presented in his videos. TDD workflow is fine; it's not thinking about the pink elephant (the source code) idea that bugs me.

Robert Martin is author of Agile manifesto. https://www.quora.com/Why-does-Kent-Beck-refer-to-the-redisc... The original description of TDD was in an ancient book about programming. It said you take the input tape, manually type in the output tape you expect, then program until the actual output tape matches the expected output.

+1. TDD could be considered as a derivation of the Scientific Method (Hypothesis Testing).

https://en.wikipedia.org/wiki/Scientific_method

https://en.wikipedia.org/wiki/Hypothesis

Test first isolates out a null hypothesis (that the test already passed); but not that it passes/fails because of some other chance variation (e.g. hash randomization and unordered maps).

https://en.wikipedia.org/wiki/Null_hypothesis

... https://en.wikipedia.org/wiki/Test-driven_development

Re: TDD Doesn't Work

#115

Post by westurner »

jacques_chester wrote: post #78
bitL wrote: post #44

I thought that TDD morphed into ending up with a regression/integration/conformation test suite instead of using tests as specifications written prior to writing products. And even 100,000s of tests won't help you in very advanced applications like cloud/cluster infrastructure as sometimes it's simply too difficult if not impossible to come up with tests (imagine observer effect when your cluster deadlock happens onl…

How do the people who write them know that they work?

> How do the people who write them know that they work?

Test first isolates that a given test doesn't already pass (without any additional code).

Test after (but before committing) also seems to require a more thorough critical analysis.

And then someone finally fuzzes the code.

Re: TDD Doesn't Work

#116

Post by westurner »

digibo wrote: post #31
aikah wrote: post #9

Earlier quoted context omitted.

Aren't tests supposed to be a tool to help design API? in that perspective a test should be written first. The problem IMHO is the choice of methodology as there is several kind of tests. Some may be more time consuming when it comes to the set up.

Personally, I think unit tests shine best when you're designing an API. I can swing from hate to love and back about TDD in minutes, but when it comes to thinking about how your code will be used, unit tests (did we stop using that term?) are a tremendously useful tool I have. I guess if all code written could be seen as an API, TDD would be great, but that's not the world I live in.

> I guess if all code written could be seen as an API, TDD would be great, but that's not the world I live in.

If not an "Application Programming Interface", isn't all code an Interface? There's input and there's output.

With Object Oriented programming, that there is an interface is more explicit (even if all you're doing is implementing objects that are already tested). There are function call argument (type) specifications (interfaces) whether it's functional or OO.

Re: TDD Doesn't Work

#117

Post by cocktailpeanuts »

justicezyx wrote: post #99
cocktailpeanuts wrote: post #95

Earlier quoted context omitted.

There are many other large companies other than Amazon and Google. Just because Google and Amazon don't have QA team doesn't mean the occupation doesn't exist.

I do not intend to say QA as an type of engineering position does not exist. What I was responding to is the parent's claim that "most large companies have a QA team so this fear is not super tangible to many developers"; that's not true in general.

It's kinda funny I'm even debating about this, I don't know where your confidence comes from.

You use the two most edge cases as an example (Amazon and Google). Most other "less techy" companies don't have the luxury to not run QA. Good for you that Google and Amazon doesn't have QA, but those are the exception, not the rule. Just go to glassdoor and search for QA and you'll see tons of QA job positions for large companies.

Since you mention Amazon, for example, WalMart has QA engineer positions.

Re: TDD Doesn't Work

#118

Post by bitL »

cocktailpeanuts wrote: post #117
justicezyx wrote: post #99

Earlier quoted context omitted.

I do not intend to say QA as an type of engineering position does not exist. What I was responding to is the parent's claim that "most large companies have a QA team so this fear is not super tangible to many developers"; that's not true in general.

It's kinda funny I'm even debating about this, I don't know where your confidence comes from. You use the two most edge cases as an example (Amazon and Google). Most other "less techy" companies don't have the luxury to not run QA. Good for you that Google and Amazon doesn't have QA, but those are the exception, not the rule. Just go to glassdoor and search for QA and you'll see tons of QA job positions for large com…

Google used to have SETs, likewise MS used to have SDETs, but they dumped them in favor of DevOps which are the new QAs.

Re: TDD Doesn't Work

#119

Post by justicezyx »

cocktailpeanuts wrote: post #117
justicezyx wrote: post #99

Earlier quoted context omitted.

I do not intend to say QA as an type of engineering position does not exist. What I was responding to is the parent's claim that "most large companies have a QA team so this fear is not super tangible to many developers"; that's not true in general.

It's kinda funny I'm even debating about this, I don't know where your confidence comes from. You use the two most edge cases as an example (Amazon and Google). Most other "less techy" companies don't have the luxury to not run QA. Good for you that Google and Amazon doesn't have QA, but those are the exception, not the rule. Just go to glassdoor and search for QA and you'll see tons of QA job positions for large com…

I do not want to nit picking our discussion. But I seem failed to convey my ideas: 1. QA exists prevalently 2. In general software companies do not have dedicated QA team for every engineer team 3. Lack of testing is a real concern for maintainability in big companies, because developers need to write tests themselves.

I do not disagree with your examples, but they are orthogonal to my point #2 #3. Let's go back to your original comment:

""" The thing is, most large companies have a QA team so this fear is not super tangible to many developers. And small startups are more focused on building stuff quickly (which they should be). """

The comment was responding to: """ Once you release a product that will be used by many customers and developed by many people throughout its lifecycle, which come and go as the time passes, you won't be able to maintain/extend it without a proper testing suite. It's not only about complexity, but also about maintainability. Some tests will also rot in time. """

My understanding is that, you meant to say that the fear of lack of testing harms maintainability is not so relevant to developers in big companies, because they have dedicated QA team (writ tests for them).

My comment says, in general, big companies do not have dedicated QA teams for dev teams, so there is no dedicated QA team write tests for dev team. My examples are 2 of the largest software companies in the world. Among them, Google redefined how people access information, Amazon reinvented how developers access computing resources.

I think my examples support my intention to prove that your statement and what it implied, in general, is not true.

Re: TDD Doesn't Work

#120

Post by lisivka »

bitL wrote: post #111
lisivka wrote: post #107

Earlier quoted context omitted.

Each test case increases confidence in the system by small amount. Confidence never can approach 100% (because we need to predict future to achieve that), so no amount of testing can give you 100% confidence, only 80%, 95% (2x price), 98% (4x), 99,5% (8x), 99,95% (16x), 99,995% (32x), and so on. It's your message, right?

My message is more like your confidence after writing hundreds of thousands tests might be just 50%. From my own experience, every single bad case that can happen in a complex system will happen at some point at some customer, wrecking their system and costing them potentially millions, in serious trading bugs even leading to a bankruptcy. Your testing suite won't catch these initially but reactively when you add tha…

In such cases, I use "torture" test cases: lengthy, random test cases, which are trying to abuse and overload system with no data, incorrect data, random data, huge data, high latencies, duplicated messages, missed messages or random aborts, random speaks, etc. They allows me to discover situations not covered by test cases. I also try to use underpowered hardware for such testing. Of course, I cannot imagine all possible torture scenarios, but I saw lot of bug and security reports, so I still know lot of scenarios, more than I willing to write tests for.
Post reply on HN