100,000 e2e selenium tests? Sounds like a nightmare • Hacker News Board

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#31

Post by brown9-2 » Wed, May 14, 2014, 3:41 PM UTC

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

Typically this would be answered in a unit test like this:

This would be a really bad way to test all of "create a user, assign the user as an administrator, log off of the root admin, log in as the new user, create a product, delete a product, get the product list and verify the product isn't on the list".

You would be ignoring 80% of the flow.

The goal of unit testing is to test each piece in isolation and to provide confidence that when you have to assemble a lot of little pieces of code to form a larger piece of functionality, that each of those little pieces is working as it's contract says it should.

If your integration test containing 8 steps failed, would you know which piece of the puzzle caused the failure from the integration test report itself?

Of course, you could go the route of testing both small pieces of functionality and also testing large pieces (the integration tests), so that you can have overlapping levels of confidence.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#32

Post by Retric » Wed, May 14, 2014, 3:46 PM UTC

jameskilton wrote: post #26

Retric wrote: post #21

Earlier quoted context omitted.

That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.

I don't see how your definition of refactoring and mine differ.

In your definition if you introduce or remove a bug your not refactoring. More basically for things like simulations or graphics you may trade accuracy for speed so your code may behave slightly differently, but be much faster. The important bit with refactoring is the goal is not fixing a specific bug, however realistically for any sufficiently large change bugs can and will be both added and removed.

Edit: "Typically, refactoring applies a series of standardised basic micro-refactorings, each of which is (usually) a tiny change in a computer program's source code that either preserves the behaviour of the software, or at least does not modify its conformance to functional requirements." http://en.wikipedia.org/wiki/Code_refactoring

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#33

Post by jlu » Wed, May 14, 2014, 3:48 PM UTC

jameskilton wrote: post #28

jlu wrote: post #22

Earlier quoted context omitted.

@jameskilton you seem to be very insightful in regard of testing, mind if I ask a couple of questions in the future privately ? Thanks.

I'm always happy to discuss testing! My personal email is in my profile.

Thanks, much appreciated.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#34

Post by jameshart » Wed, May 14, 2014, 3:49 PM UTC

lmm wrote: post #19

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

Does deleting products ever break? I wouldn't bother with any tests, unit or functional, for a code path that doesn't have any logic in it - if it compiled then it's almost certainly correct. Where testing is useful is when there's complex logic. And such logic is much easier to test at the unit level.

In most real systems, deletes of core business objects like 'product' are really only soft deletes; there are caching layers; deleting things has knock-on consequences for other things (if you delete a product when an instance of it is in a customer's shopping basket, what happens?) - it doesn't seem illogical to want to test that when you 'delete' a product, it disappears from the product catalog... but then you'd want to do other tests to verify it remains in a shopping basket, and is still visible in purchase history, and so on.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#35

Post by lmm » Wed, May 14, 2014, 3:54 PM UTC

jameshart wrote: post #34

lmm wrote: post #19

Earlier quoted context omitted.

Does deleting products ever break? I wouldn't bother with any tests, unit or functional, for a code path that doesn't have any logic in it - if it compiled then it's almost certainly correct. Where testing is useful is when there's complex logic. And such logic is much easier to test at the unit level.

In most real systems, deletes of core business objects like 'product' are really only soft deletes; there are caching layers; deleting things has knock-on consequences for other things (if you delete a product when an instance of it is in a customer's shopping basket, what happens?) - it doesn't seem illogical to want to test that when you 'delete' a product, it disappears from the product catalog... but then you'd w…

Fair enough - but in that case we're really not talking about the kind of test the grandparent described, the one that looks like

    p = productDao.create("My New Product")
    assert true p.delete()

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#36

Post by opendais » Wed, May 14, 2014, 4:30 PM UTC

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

I agree with you.

In my experience, the value in unit tests is in refactoring. I'd only write them as I refactor [to confirm the functionality matches before/after the refactor].

Most of the modifications we do to common libraries, etc. involve higher level rewrites that would break the unit tests anyway and the only way to confirm things still function is integration tests. :/

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#37

Post by briantakita » Wed, May 14, 2014, 4:51 PM UTC

> but I’d much rather replace those with even higher level system tests through Capybara or similar

I've been doing this for years. It's been working out very well. I'm glad test abstraction scope is being talked about now.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#38

Post by briantakita » Wed, May 14, 2014, 4:54 PM UTC

columbo wrote: post #2

> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…

> I've gone the salesforce route, instead of using selenium all my "unit tests" are executed against the rest api.

I've taken the same route for apis and for client side testing.

On my current project, I'm creating a rich client. I have an extensive jasmine test suite with edge case and race condition testing.

I created a library called jasmine-flow, to organize the tests into flows. This allows edge case testing while reducing duplicate test setup. In my case test suite time was reduced by 10x.

https://github.com/btakita/jasmine-flow

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#39

Post by hugs » Wed, May 14, 2014, 4:59 PM UTC

Selenium creator and Sauce founder here. Not enough automated e2e is also a nightmare. (Ask me about my experience helping to fix HealthCare.gov sometime!) As usual, the right answer is: a little bit of every kind of test, depending on the risks involved. The article starts (and ends) quite hyperbolic, but comes to the same "everything in moderation" conclusion.

Re: 100,000 e2e selenium tests? Sounds like a nightmare

#40

Post by noir_lord » Wed, May 14, 2014, 5:09 PM UTC

dasil003 wrote: post #5

> Now if we could just convince SalesForce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change... I agree with the overall thrust of the article. Unit testing is indispensable, especially for me as a rubyist where there is basically nothing checked before runtime. But I find this conclusion that SalesForce is "doing it wrong" to be just as arrogant a…

> But I find this conclusion that SalesForce is "doing it wrong" to be just as arrogant and presumptuous as DHH's blustery approach.

Considering this is a business that is spending vast sums of money on doing this I think you are spot-on, yes we all joke about businesses throwing good money after bad but someone had to sit down and defend this approach.