Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

100,000 e2e selenium tests? Sounds like a nightmare

watirmelon.com

Post reply on HN
Print view View original

51–58 of 58 posts

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#51

Post by twic »

briantakita wrote: post #37

> but I’d much rather replace those with even higher level system tests through Capybara or similar I've been doing this for years. It's been working out very well. I'm glad test abstraction scope is being talked about now.

Oh cripes, me too! And i'm glad that there's a genuine plurality of views.

Test scope has been talked about for years, but almost always in the form of sermons by fervent believers in unit testing, castigating those who dare to lean more heavily on system and integration tests for their dangerous heresy.

Occasionally, these sermons admit the existence of a "testing pyramid", but take note of the necessary shape of that edifice: it's unit tests, but with a decorative cap of other kinds of test inexorably dwindling in volume as they reach higher levels. I'm not interested in this all this alleged pyramid science. Talk to me about testing towers, testing dolmens, then we can have a genuine conversation.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#52

Post by twic »

This article manages to spend quite a lot of time talking about its author's hurt feelings before it gets to anything that resembles an argument. When it does, it's this:

The real problem with end to end tests is that when end to end tests fail, most of the time you have no idea what went wrong so you spend a lot of time trying to find out why.

I've worked on big systems with predominantly (in some cases, exclusively) high-level test coverage. I can confirm that this indeed a thing. Unit tests very rarely fail in such a way that the cause of the failure is not immediately obvious. High-level tests often do. Not most of the time, but often.

Here's the thing, though. When a high-level test fails without an obvious cause, that is something that unit tests would not have caught at all.

Obvious causes are things that are localised defects in individual parts. Your product listing is showing in the wrong order? The error is in the sorting code. Orders in Japan are being billed with prices off by a factor of about a hundred? The error is in the currency conversion code. Unit tests would catch those errors and make their location blindingly obvious. High-level tests catch them, and make their location sufficiently obvious; obvious enough that pinpointing the problem is not a significant speed bump.

Non-obvious causes are things that are defects in interactions between multiple, often distant, parts. Only every second FTP'd file is triggering emails to users? Turns out the error handling code in the parser is disabling the scheduled refresh checking job, which is then not triggering the batch job to recalculate the numbers, which is then not feeding new data to the email job. Every single part is working as intended, but the composition of those parts is not. That's a tough problem, and when the high-level test catches it, you will have hours and hours of the wrong kind of fun unravelling the strands of cause and effect to pinpoint the problem. But unit tests would not have caught it at all.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#53

Post by columbo »

lmm wrote: post #19
columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

Does deleting products ever break? I wouldn't bother with any tests, unit or functional, for a code path that doesn't have any logic in it - if it compiled then it's almost certainly correct. Where testing is useful is when there's complex logic. And such logic is much easier to test at the unit level.

Actually yes! The gist of the story was deleting a product was not removing it from the product list because of query caching. It was something you would only have found when running against the site itself as query caching doesn't exist at unit or integration testing time. That was an interesting one. Our rest layer tests found it, although even still it took awhile to figure out (how the #@!$ is it still in the list?)

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#54

Post by ffreire »

hugs wrote: post #47
ffreire wrote: post #46

Earlier quoted context omitted.

I'd actually be extremely interested in hearing more about your experiences with HealthCare.gov. I'm working on implementing test infrastructure for my current employer and would love to learn from folks more experienced than myself about the process!

Thanks for taking the bait. :-) I don't have time to write about it today, but hope to someday. Since this won't be the last time the government creates a website, the government needs to get good at this stuff. The tech team did an amazing job rescuing the site, and I hope the geeky lessons (especially from the testing perspective) get shared someday.

I'll keep my eyes peeled, then!

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#55

Post by Retric »

twic wrote: post #49
Retric wrote: post #21

Earlier quoted context omitted.

That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.

The standard definition of refactoring is that it doesn't change behaviour. As Martin Fowler put it: Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure. You correctly identified a consequence of this in your later comment: > In your definition if you introduce or remove a bug your not refactoring. That is…

The term refactoring dates back at least to the 80's well before TDD showed up and it was used as a shorthand for cleaning up code without the focus on tiny changes. More importantly bugs are often related to side effects such as how long a method takes as such using even one more or one less cycle any any code path prevents code from being 'pure' in your definition as it may add or remove a bug as would changing it's memory footprint etc.

Granted, that may seem pedantic but if you look closely you realise nobody uses your 'ideal' definition in practice.

PS: Feel free to use / introduce new terms such as 'Pure refactoring' but understand they don't change what refactoring actually means. As to popularity wikipedia's "does not modify its conformance to functional requirements" suggests that's the commonly understood definition.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#56

Post by jameshart »

lmm wrote: post #35
jameshart wrote: post #34

Earlier quoted context omitted.

In most real systems, deletes of core business objects like 'product' are really only soft deletes; there are caching layers; deleting things has knock-on consequences for other things (if you delete a product when an instance of it is in a customer's shopping basket, what happens?) - it doesn't seem illogical to want to test that when you 'delete' a product, it disappears from the product catalog... but then you'd w…

Fair enough - but in that case we're really not talking about the kind of test the grandparent described, the one that looks like p = productDao.create("My New Product") assert true p.delete()

The grandparent was pointing out that a unit test like that doesn't give you any confidence in your system precisely because it fails to exercise the broader context - so while it tells you that 'yup, you deleted it', it doesn't tell you that 'other code is now treating it as deleted'. GP was arguing that a unit test was not useful but a functional test was.

You, on the other hand, argued that simple deletion didn't need ANY tests, unit or functional - I was responding with examples that demonstrated that functional integration tests for deletion are perfectly valid.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#57

Post by brazzy »

jameskilton wrote: post #17
brazzy wrote: post #15

Earlier quoted context omitted.

The problem is that refactoring definitely can also cause components to not do what they're supposed to do outside the happy path.

Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).

Well, then by that definition you are not refactoring either, and in fact nobody is refactoring. Because any change can unintentionally change the behavior. That's what we call a bug, and them bugs don't care about your definition.

And Unit tests alone cannot ensure that the non-happy-paths are functional.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#58

Post by lmm »

jameshart wrote: post #56
lmm wrote: post #35

Earlier quoted context omitted.

Fair enough - but in that case we're really not talking about the kind of test the grandparent described, the one that looks like p = productDao.create("My New Product") assert true p.delete()

The grandparent was pointing out that a unit test like that doesn't give you any confidence in your system precisely because it fails to exercise the broader context - so while it tells you that 'yup, you deleted it', it doesn't tell you that 'other code is now treating it as deleted'. GP was arguing that a unit test was not useful but a functional test was. You, on the other hand, argued that simple deletion didn't…

You need to compare apples to apples. An integration test that tests some logic is better than a unit test that doesn't test any logic, duh. But if you're not testing any logic then the test is useless either way. And if you are testing something like cache invalidation, then you can do that just as well, probably better, at the unit level.
Post reply on HN