100,000 e2e selenium tests? Sounds like a nightmare • Hacker News Board

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#21

Post by Retric » Wed, May 14, 2014, 3:08 PM UTC

jameskilton wrote: post #17

brazzy wrote: post #15

Earlier quoted context omitted.

The problem is that refactoring definitely can also cause components to not do what they're supposed to do outside the happy path.

Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).

That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#22

Post by jlu » Wed, May 14, 2014, 3:09 PM UTC

jameskilton wrote: post #17

brazzy wrote: post #15

Earlier quoted context omitted.

The problem is that refactoring definitely can also cause components to not do what they're supposed to do outside the happy path.

Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).

@jameskilton you seem to be very insightful in regard of testing, mind if I ask a couple of questions in the future privately? Thanks.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#23

Post by lmm » Wed, May 14, 2014, 3:10 PM UTC

pfg wrote: post #20

lmm wrote: post #19

Earlier quoted context omitted.

Does deleting products ever break? I wouldn't bother with any tests, unit or functional, for a code path that doesn't have any logic in it - if it compiled then it's almost certainly correct. Where testing is useful is when there's complex logic. And such logic is much easier to test at the unit level.

I can imagine deleting products may be failing due to some UI change. Maybe a button isn't visible in certain IE versions. Maybe the JS code executing the backend request is failing in some browsers. That's where e2e tests are useful.

Those are purely failures in the UI layer. To the extent that "unit testing" a web frontend is possible, you could catch those problems with a pure unit test that mocked out the backend, it wouldn't have to be an end-to-end test.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#24

Post by dreamfactory2 » Wed, May 14, 2014, 3:16 PM UTC

jameskilton wrote: post #4

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything works together. Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor you…

> TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!)

Interesting. I think this is probably right and also the reverse of what most people try to do (they start with units and work up)

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#25

Post by matlock » Wed, May 14, 2014, 3:17 PM UTC

I think this shows a different understanding than for example we follow at Codeship.

Tests aren't there to test our code, but to make our Users happy. e2e tests make sure we don't break features and are able to validate new features don't kill the most common workflows. Thus we can innovate quickly, build new features and refactor. This makes our users happy. Unit tests, controller tests or others are mainly an optimisation as writing an e2e test for every failure state is not an option.

It's a different understanding of why we test, but makes sure we focus on the right things with it.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#26

Post by jameskilton » Wed, May 14, 2014, 3:18 PM UTC

Retric wrote: post #21

jameskilton wrote: post #17

Earlier quoted context omitted.

Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).

That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.

I don't see how your definition of refactoring and mine differ.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#27

Post by falcolas » Wed, May 14, 2014, 3:21 PM UTC

jameskilton wrote: post #4

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything works together. Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor you…

In my experience, if your integration testing is only happy path testing, you're doing integration testing wrong.

Integration testing should, at a minimum, include:

    * Combinatorial testing
      * i.e. testing all possible input combinations for every pair of two inputs
    * Fuzz testing
    * Limit testing
    * Success path testing
    * Failure case testing

Integration testing will discover more component interaction bugs than unit testing alone, and doing only happy path testing is doing it wrong.

On the other hand, I wholly believe that testing done by the developer alone will never catch as many bugs as testing done by both the developer and a QA engineer to the mix will. They think differently than we do, and as such are adept at finding our blind spots.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#28

Post by jameskilton » Wed, May 14, 2014, 3:21 PM UTC

jlu wrote: post #22

jameskilton wrote: post #17

Earlier quoted context omitted.

Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code. Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).

@jameskilton you seem to be very insightful in regard of testing, mind if I ask a couple of questions in the future privately ? Thanks.

I'm always happy to discuss testing! My personal email is in my profile.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#29

Post by ryanjshaw » Wed, May 14, 2014, 3:28 PM UTC

dreamfactory2 wrote: post #24

jameskilton wrote: post #4

Earlier quoted context omitted.

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything works together. Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor you…

> TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!) Interesting. I think this is probably right and also the reverse of what most people try to do (they start with units and work up)

I find this approach has a number of advantages:

- it forces you to flesh out your conceptual/paper design in a deliverables-oriented way and makes it obvious where the gaps are, which gives you a good idea of how ready you are to wrap up design and start development, and in turn can be used to create your work breakdown structure and task prioritisation; sometimes it can result in identifying simplifications upfront (saving you time down the line)

- when you have a good idea of the scope and complexity of a solution like this, you're less likely to waste time on the less important stuff

- if you're the tech lead on a team, giving your developers the high-level components and tests means they're more likely to get things right (or at least limit the damage they do when they get things wrong)

Without component-level unit tests or design, you can easily dive into a solution without thinking it through and really regret it.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#30

Post by jameskilton » Wed, May 14, 2014, 3:30 PM UTC

falcolas wrote: post #27

jameskilton wrote: post #4

Earlier quoted context omitted.

No-one ever said that integration tests and unit tests are mutually exclusive. Integration test the happy paths, small integration suite that proves everything works together. Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor you…

In my experience, if your integration testing is only happy path testing, you're doing integration testing wrong. Integration testing should, at a minimum, include: * Combinatorial testing * i.e. testing all possible input combinations for every pair of two inputs * Fuzz testing * Limit testing * Success path testing * Failure case testing Integration testing will discover more component interaction bugs than unit te…

We may have a wording definition issue here. I've heard multiple definitions of "integration tests". I usually use them where some people use "acceptance tests", aka, top-down be-a-user type testing (selenium in Firefox, for example). Sorry if there was any confusion there.

Agreed having a QA team also go through the app is great for finding things that tests can't catch, like usability errors, or crazy edge cases you didn't think about, etc. I have gotten to work with a good QA team and it's amazing the things they find.