Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Giving up on test-first development

iansommerville.com

Post reply on HN
Print view View original

181–190 of 224 posts

Re: Giving up on test-first development

#181

Post by hinkley »

ekidd wrote: post #5

From the article: Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests. Interesting. I've often found that the lack of tests leaves me absolutely terri…

See, I find the reverse to be true.

I too don't have the disposition to patiently wade through fixing tons of tons of broken tests over and over again. IF I have to I can do it, but I know better than to unreservedly trust my judgement about how things are progressing.

But what this has taught me is to stop trying to push water uphill. If testing is hard because of questionably coupled code, refactor it NOW instead of waiting for things to get bad. The refactoring almost always suggests new features we could add to the code (or makes me backpedal on pronouncements that certain things were 'impossible'), and the number and kind of tests that break is reduced.

I came across a quote recently from Bertrand Meyer (the Design By Contract guy), where he suggested that code for making decisions and code for acting on those decisions should be separated. I found myself nodding along to this because it was something I knew intuitively but had never articulated: Decisions without actions don't need mocks, and actions without decisions need at most one or two (often none). Decisions can be tested (possibly in parallel) with unit tests, and actions tested with functional tests. Then all your integration testing is that decisions lead to actions (basically that event dispatch works) and that catastrophic failures are handled gracefully. A proper testing pyramid.

Now I want to figure out which of his books or interviews this was in because I want to read the rest of what he had to say on the subject.

Re: Giving up on test-first development

#182

Post by hinkley »

kayoone wrote: post #131
BonsaiDen wrote: post #72

Earlier quoted context omitted.

For smaller projects I definitely agree with having both unit and integration tests, especially for libraries. One thing to look out for is the fact that you can always "cheat" in unit tests, e.g. you can be "lazy" and set up some internal state directly in the test to skip huge amounts of initialization, this of course becomes a problem when there's no actual integration test for making sure that the exact state can…

yeah, well you just test what you own anyway, for other stuff you could just grab some kind of dummy response and use that for testing your manipulation of that data, but of course you have to trust your customers endpoints to return the data in the correct format because that is out of your control.

Trust, but verify.

We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.

Re: Giving up on test-first development

#183

Post by hinkley »

schrodinger wrote: post #91
cousin_it wrote: post #37

Earlier quoted context omitted.

Yeah, I mostly prefer end-to-end tests as well. Though to be fair, they are often slower than unit tests, because you need to start up the whole system. And they are worse at pinpointing problems, though that doesn't seem to be a big deal in practice.

I like to test the whole system via end-to-end tests, as they're the best bang for the buck. And then I'll create unit tests for more algorithmic code, like a parser, sort algorithm, shortest path calculator, financial calculations. Those also tend to require the least amount of test context setup, making them less painful to write.

I have almost always worked for customers who enjoyed changing their minds arbitrarily and often in ways they swore they would never do.

In the face of grossly changing requirements, I've never had much luck keeping E2E tests up and functioning properly. And people have a bad habit of investing more time and energy than a particular test is worth in trying to keep it working or porting it to the new requirements.

Unit tests are cheap. If the requirements change invalidates twenty of them, you just delete them and write new ones. Easy.

Re: Giving up on test-first development

#184

Post by jmathes »

kyberias wrote: post #166
jmathes wrote: post #157

TDD is best when you're writing code that talks to other code. So APIs, database models, etc. Pure functions, and code that has dependencies you can inject and mock. You should never abandon TDD in situations like this. It's true that it's harder to write TDD for code with side effects or that draws UI. It doesn't really make sense to use TDD for this. You shouldn't conflate the two. Also, "always pass the majority o…

You state that "TDD is best" in certain scenarios but fail to provide explanation. Why do you think TDD is best in the situations you enumerated? In fact, how does "database model" talk to other code? I'm pretty sure unit tests, automated tests and continuous integration existed 8 years ago in 2008. According to wikipedia, CI was named by Grady Booch in 1991.

I think TDD is best in the situations I enumerated because writing tests for single behaviors of single functions is easy, fast, forces you to discover errors that cause your tests to trivially test, and forces you to discover bugs in your understanding of the problem.

Continuous "Deployment", I guess. The term may have existed before I realize, but it certainly wasn't in popular use before this:

http://timothyfitz.com/2009/02/10/continuous-deployment-at-i...

Re: Giving up on test-first development

#185

Post by lfowles »

crdoconnor wrote: post #174
lfowles wrote: post #159

Earlier quoted context omitted.

That's assuming it works correctly the first time (I haven't had that experience often, even for "trivial" code :( ). Even for "run once" functions, I still use a few tests to develop them and make sure my expectations are correct. With a good framework, setting up a handful of unit tests takes just about as much dev time as running the function in a REPL.

Ok, so let's say you were experimenting with the selenium library to see if you could scrape what comes out of skyscanner. What steps would you take?

I'm not clear on what the scope of selenium is or how it works, but for a general web scraper, I'd identify some targets I want out of skyscanner. Here's a quick googletest butchery for "I want to make sure my function returns flights for a known good flight search."

    TEST(SkyScanner, ListFlights)
    {
        flights = MyFlightScrapingFunction(LAX, AMS, date+1 month, 1 way, 1 adult)
        EXPECT_THAT(flights, SizeIs( Gt(0) )

        EXPECT_THAT(flights, Each( AllOf( 
                                       Field(&Flight::from, Eq(LAX)), 
                                       Field(&Flight::to, Eq(AMS)),
                                       Field(&Flight::seats, Eq(1)) 
                                         )))
    }
I feel that anything simpler ("is this possible with selenium?") is a documentation moment.

Re: Giving up on test-first development

#186

Post by hinkley »

mbrock wrote: post #57
sago wrote: post #27

Earlier quoted context omitted.

> that tells us that we wrote a bad test, not that TDD is bad I strongly disagree. The only thing that matters to how software works for an end user is its boundary. You simply can't do TDD by only testing the boundary. You have to write a test before each unit of code, so you have to test those internal interfaces, the way units of code work. But that is exactly the stuff that should be allowed to change. It's a mis…

Changing tests is work, yes. Unit tests are precisely about verifying implementations, and maintaining them is the price you pay for having these layers of verification. It's inertia in the way a safety net is a barrier.

I have always found the 'safety net' analogy uninspiring, and people who think they're tough and manly don't need safety nets, thankyouverymuch.

Now, safety equipment in a race car is something else entirely. Half of it keeps you from dying immediately on impact, sure, but the other half actually lets you go faster. The 5 point harness, for example, helps keep you from losing control of the vehicle during high G maneuvers, by keeping your torso in the seat and therefore your legs and arms in (roughly) the correct orientation to the steering wheel and the pedals.

Re: Giving up on test-first development

#187

Post by douche »

catnaroek wrote: post #120
barrkel wrote: post #40

Earlier quoted context omitted.

The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings. People "know it when they see it" without necessarily being able to define it. So you write the code; expose it to a preference function / oracle, iterate, and hill-climb towards a local optimum. The revealed preferences may give enough information to make a creative leap off the local hill ont…

> The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings. Is this preference function implemented on a computer, or is it just a person giving you seemingly random answers in response to whatever you tell them? > People "know it when they see it" without necessarily being able to define it. That path leads to hell. > After a solution has been found,…

Where is the logic in most business logic? \s

Seemingly random answers, which are not functionally pure (aka ask the same question, get the same answer) are par for the course

Re: Giving up on test-first development

#188

Post by hinkley »

maccard wrote: post #113
BonsaiDen wrote: post #52

Earlier quoted context omitted.

I in part agree with the pinpointing, though in my experience this really boils down to an issue of scope and how much of the data you want to test in each of your tests. E.g. an API returning user data, do you have one test for the whole set of data or one test per field. For our use case we have some pretty "fancy" deep-equals logic for nested structures and allow to specific fields as "to be ignored" in our test e…

Count yourself lucky that your test takes 10 seconds. A partial build on my project takes ~3 minutes to compile + link. A full build is about 3 hours for everything.

I had a build that took 25 hours to run once. Never again. The only reason we got any work done is that we ran a 1, 3 and 19 hour part of it in parallel. But I really just wanted to throw out the long part (bad E2E tests) and start over.

Re: Giving up on test-first development

#189

Post by lgunsch »

kazinator wrote: post #117

TDD: Write the test case representing "this USB host controller driver has no race conditions", and then just fill in the code, and out pops a race-free USB host controller driver! (Of course, it does nothing so far other than demonstrating freedom from races, but that's just a small matter of writing more tests for actual USB requirements and fulfilling those.)

TDD is not about reduction of bugs, or race conditions. Robert Martin has said that reduction of bugs is not sufficient enough to warrant using TDD for code.

TDD is about achieving better designed code that is maintainable and readable for many years. TDD uses unit tests, not integration tests. So, the behaviour of each function is asserted independently. Maybe you have a function that sets up a data structure at a particular memory location. Your unit test then is just to assert that the data structure at the memory location was setup correctly. You should even be able to unit test functions used in a bootloader like Grub.

Re: Giving up on test-first development

#190

Post by barrkel »

catnaroek wrote: post #120
barrkel wrote: post #40

Earlier quoted context omitted.

The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings. People "know it when they see it" without necessarily being able to define it. So you write the code; expose it to a preference function / oracle, iterate, and hill-climb towards a local optimum. The revealed preferences may give enough information to make a creative leap off the local hill ont…

> The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings. Is this preference function implemented on a computer, or is it just a person giving you seemingly random answers in response to whatever you tell them? > People "know it when they see it" without necessarily being able to define it. That path leads to hell. > After a solution has been found,…

> > People "know it when they see it" without necessarily being able to define it.

> That path leads to hell.

That's why they pay us the big bucks.

If people could define their problems well enough without iterative design, waterfall development would be a success story and all software would be outsourced to the lowest bidder.

Post reply on HN