Live data from Hacker News

Giving up on test-first development

iansommerville.com

211–220 of 224 posts

Re: Giving up on test-first development

#211
post #12

One of the things that people aim for in writing tests is orthogonality - different tests should not break for the same reason. This promotes the ability to refactor and change your code. I have also seen massive codebases, with masses of tests which were rarely run, and which effectively concreted the code and stopped it from changing.

But doesn't test redundancy reduce the risk of a test isn't testing the thing you thought it was? If the same situation is tested in 2 different ways, a bug in one test might cause the test to fail to correctly handle all case, but the second test might still catch. Maybe auto-generation of test cases would be a better technology?

Typically tests can provide fault detection and fault isolation, but to different degrees.

A feature/e2e test typically provides the most effective kind of fault detection, because it combines all the components of the real system with no stubs or mocks. But if it shows a fault, then typically that fault is hard to identify, because it wasn't previously driven out by a more detailed integration or unit test.

Contrariwise, integration or unit tests will typically be best suited to isolating faults, but typically those faults are limited to the class of things you included in your tests.

This is why we have "test pyramids": a handful of slow, brittle feature tests at the top, then an increasing volume of faster, less-brittle integration and unit tests.

TDD is almost orthogonal to the testing pyramid, with one difference from non-TDD tests. In TDD each line of code was, ideally, driven out by a test or tests. Fault detection is increased, because each line was written in response to a manufactured (test-first) fault.

Re: Giving up on test-first development

#212
post #205

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar i'll take a Bazaar of 5000 artists any day over a forced Cathedral.

Testing practices are orthogonal to the cathedral vs bazaar idea. Linux is a great example of this. Anyone can checkout the code and work on it, but upstreaming the changes requires coordination and approval with the community. The lessons by Raymond are also given in the context of OSS. Unfortunately it would be extremely difficult to find a team of developers that are passionate about the kind of stuff that SAP is…

really good point about SAP! It's soooooo boring. That's the million dollar question - how to u inprise passion in a team of programmers all working for cash and not passion? I'd argue forcing TDD on your most brilliant artists is not the way. Or telling them that they are not "professional" if they don't follow what you consider to be "best practices" is a bad idea. That spark of true passion is worth it's weight in gold. Your best shot is to make an env where those sparks CAN happen vs. extinguish each little spark before it can grow.

Re: Giving up on test-first development

#213

TDD: Write the test case representing "this USB host controller driver has no race conditions", and then just fill in the code, and out pops a race-free USB host controller driver! (Of course, it does nothing so far other than demonstrating freedom from races, but that's just a small matter of writing more tests for actual USB requirements and fulfilling those.)

TDD is not about reduction of bugs, or race conditions. Robert Martin has said that reduction of bugs is not sufficient enough to warrant using TDD for code. TDD is about achieving better designed code that is maintainable and readable for many years. TDD uses unit tests, not integration tests. So, the behaviour of each function is asserted independently. Maybe you have a function that sets up a data structure at a p…

> TDD is about achieving better designed code that is maintainable and readable for many years.

It isn't entirely clear to me why this is true. Clearly, TDD forces you to think beforehand about what you want your program to do, since that's ultimately what a test suite is: an executable description of what you want your program to do. However, it doesn't necessarily follow from this that your code will be well-designed or readable. Tests are about evaluating whether an implementation conforms to a specification, not whether the design is actually good. To evaluate a design, you need performance metrics. In other words, you need an answer to the question: “Is this design actually helping me achieve my goals?”

Engineering design is somewhat of a black art. A designer has to be both organized (to formulate and carry out plans) and flexible (to reconcile goals that may conflict with each other and/or evolve over time). Having a large toolbox of methodologies and problem-solving heuristics is a good thing, but it's also important to avoid the kind of mindset where your favorite tool is the One True Tool, or your favorite problem-solving approach is the One True Approach.

Re: Giving up on test-first development

#214

Earlier quoted context omitted.

> > People "know it when they see it" without necessarily being able to define it. > That path leads to hell. That's why they pay us the big bucks. If people could define their problems well enough without iterative design, waterfall development would be a success story and all software would be outsourced to the lowest bidder.

That's wrong. It's perfectly possible to define your problem well, and still require an iterative process to find a good solution.

The requirement for an iterative process is what I mean by "oracle"; it's the thing that tells you how far off you are on this iteration, and how to change your heading for the next iteration.

Re: Giving up on test-first development

#215

Earlier quoted context omitted.

I'd translate this as "don't use tests for proof-of-concept code". At least not when success is easily observable.

In XP settings this is called "spiking" and is recognised as an entirely legitimate tactic. The point of spiking is that you don't know enough to TDD. You are outside the sweetspot where TDD is tractable. The key to spiking is that once you come with a plausible approach, or have a conceptual breakthrough, you stop, backtrack and then switch back to TDD. Like everything else in mature software engineering, agile or c…

Interesting. I've found myself drawn into XP-style delivery repeatedly, probably because I tend to work closely with clients who have rapidly shifting or unknown requirements.

Re: Giving up on test-first development

#216

Earlier quoted context omitted.

In XP settings this is called "spiking" and is recognised as an entirely legitimate tactic. The point of spiking is that you don't know enough to TDD. You are outside the sweetspot where TDD is tractable. The key to spiking is that once you come with a plausible approach, or have a conceptual breakthrough, you stop, backtrack and then switch back to TDD. Like everything else in mature software engineering, agile or c…

Interesting. I've found myself drawn into XP-style delivery repeatedly, probably because I tend to work closely with clients who have rapidly shifting or unknown requirements.

We have a London office. Email me and I can probably ask if they'll let you visit.

(They'll try to recruit you, but TANSTAAFL).

Re: Giving up on test-first development

#217

Earlier quoted context omitted.

That's wrong. It's perfectly possible to define your problem well, and still require an iterative process to find a good solution.

The requirement for an iterative process is what I mean by "oracle"; it's the thing that tells you how far off you are on this iteration, and how to change your heading for the next iteration.

That's a rather weird use of the term “oracle”, which normally means “a black box that solves a decision problem in a single step”. Normally, to decide whether you've achieved your goals, you don't use a black box - you use performance metrics that you've defined yourself (and thus can't claim not to understand).

Re: Giving up on test-first development

#218

Earlier quoted context omitted.

Trust, but verify. We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.

These have become known in Pivotal as "frenemy tests". Poke a remote API to see that a) it is running b) it hasn't dropped a world-stopping change. Typically run as a pre-build sanity check when you have remote integrations. On some Labs projects we've written extensive request tests and stub services based on documentation, then handed those to the upstream service. Usually there is some angst at this point, being t…

I still maintain F5 owes me a job. I did QA for them for a year back in the dot-com boom. The first couple of versions of BigIP didn't really support session affinity, despite being the flagship feature. I think we filed something like 6-8 pretty big bugs, all in different parts of the code.

Sadly, despite not working, they were still about 4 years ahead of their competitors.

Re: Giving up on test-first development

#219
post #21

Earlier quoted context omitted.

I've found the opposite, that TDD helps me most when the spec or my understanding of the spec is fuzzy, by forcing me to make decisions about behaviour up-front and clarify my thinking. Otherwise I can get bogged down trying to implement and specify a feature simultaneously, or spend a lot of time implementing a feature before realising I'm approaching it the wrong way.

Depending on how you work, that can be worthwhile, but I usually find I get more useful information quickly by making the decisions that make implementation easiest, getting something up on the screen that I can react to as a user (or put in front of a real user), and then seeing where that initial proof of concept falls short. In general, I've found that pushing off decisions until I have as much information as I ca…

Yes, its easier to prototype a feature first and get feedback from the users than it is to guess a load of tests to do the same thing.

Re: Giving up on test-first development

#220
post #36

Over the years I've gone from writing no tests at all, to being a die hard TDD purist, and then out the other side to writing some things with tests first, others with tests after writing the code, and some without any tests at all. In some situations I have clear view of what I need to build, and how that should work. TDD is great in that case - write a test for the expected behaviour, make it pass, refactor, rinse…

Over the years I've gone from writing no tests at all, to being a die hard TDD purist, and then out the other side to writing some things with tests first, others with tests after writing the code, and some without any tests at all. Hear, hear! Exactly the same here, for the same reasons as you and the OP mention. And observed fom a distance, it's always the same universal principle: purist behaviour (in the sense of…

"All generalizations are false, including this one." - Mark Twain
Post reply on HN