Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

131–140 of 270 posts

Re: We killed our end-to-end test suite

#131

Post by evandwight »

jdsleppy wrote: post #126
evandwight wrote: post #118

Earlier quoted context omitted.

How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.

As one example, Django handles test parallelism by creating N test databases (on the single test database server) and dividing tests into N runners. https://docs.djangoproject.com/en/3.2/ref/django-admin/#envv... You could also have multiple Docker containers running DBs.

Thank you! That's so crazy!

Re: We killed our end-to-end test suite

#132

Post by geofft »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

There are two things that can be tested here, not one: whether the upstream service conforms to the contract / API promise, and whether your code behaves correctly with respect to what the API promises.

So that gives you a number of options for testing the second one of those. Recording sample traffic and replaying it in the test suite is one approach. Actually running an instance of the service (if it's open-source - there's still value in paying someone to competently run an OSS service) in your test suite is another, as is running some clone of the service (e.g., if you're talking to S3, there are probably a hundred S3 API-compatible clones that are good enough to run in your test suite, even if, again, you are happy to pay Amazon to competently run production).

You also want to pay attention to the first one of those, but that's not a job for your test suite. That's the job for some balance between their test suite, your monitoring or production logging, and your business relationship with them.

Re: We killed our end-to-end test suite

#133

Post by tetha »

3pt14159 wrote: post #96
deckard1 wrote: post #70

Earlier quoted context omitted.

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible.

We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd have to add dev-hours on top of the E2E costs to build and maintain mountains of E2E tests.

And this has exposed really nasty bugs in common paths already, just by employees using the system.

Re: We killed our end-to-end test suite

#134

Post by hinkley »

shakezula wrote: post #127
hinkley wrote: post #124

Earlier quoted context omitted.

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

> Often E2E tests have such sunk costs involved that they materially affect the project roadmap. I've seen this first hand. When the e2e tests take hours to run, are flaky on a good day, and are only really understood by one or two people on the whole team, they can be a major roadblock to new features or even just moderate refactors.

That definitely happens. The E2E tests tend to make assumptions about how the app works (encode not just the requirements but also the architecture) and some features change the design. In order to add this feature we have to fix dozens of other tests. I’ve seen people on multiple projects team up to fix these, take over a day working together, and still not be done. They always try to tweak the tests but the test assumptions fight them.

Meanwhile if we add a feature that invalidates a unit test, you just delete the unit test and start over. Unit tests are cattle, E2E tests are pets.

Re: We killed our end-to-end test suite

#135

Post by catern »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

Your comment matches my experience very well. I had the same experience as GP and OP with low-quality e2e tests at my job. I got fed up four years ago, started something new from scratch, and now I'm the BDFL you mentioned, for a bunch of teams working in a common testing framework.

The main thing is indeed enforcing high quality standards even when individual engineers aren't very invested. You've identified some good practices right in your post, but it can take some time for people to learn these principles. And they can be reluctant if they see it as a waste of time. "These are just tests, I need to do my real work!"

For me, the crucial thing here is to avoid building things that are just for testing. If you tell someone that sleeping here is not good enough, and they need to build something more elaborate - then it's much more compelling if you can figure out how to build that so it's not just useful for a test, but also useful in production. This can be things like more flexible configurations, recovery tools for emergencies, new monitoring scripts and systems... all kinds of stuff.

If you stay focused on building things that are flexible enough to be used for both testing and production, then your life gets harder in some ways, but you can be much more strict about requiring high-quality work.

(btw, I'm hiring for the team building this infrastructure: http://catern.com/tsint_job.html )

Re: We killed our end-to-end test suite

#136

Post by sowbug »

We've seen this before. At first, the E2E tests automate a bunch of tedious manual tests. Then they become the team's "automated test suite." Then nobody (except true believers) tests their code anymore, because "our automated test suite" will surely catch any problems. Then everything starts breaking down the way the author cites.

The E2E test suite needs to be thought of differently from a "test suite." It's the last safety net to disprove that the build is worth [manual testing | dogfooding | beta release | prod release]. Any bug found there should be worth a postmortem -- even one as short as "oops, new guy forgot to update the unit tests! fixed, won't happen again!" Of course, bugs will get through to the safety net, even in a system that's working well. But they should always trigger the question whether and how the bug could have been caught earlier.

Re: We killed our end-to-end test suite

#137

Post by Daishiman »

marcosdumay wrote: post #114
munificent wrote: post #75

Earlier quoted context omitted.

> is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code. Another way to say this is that efficient e2e tests require significant continuous investment in top-tier engineer time. The question then is how much engineering time is worth being spent in that way. It may be that, yes, you can h…

> The question then is how much engineering time is worth being spent in that way. Well, since it brings more value than testing in a lower level, I would say, more than any other kind of test (except, maybe, for monitoring). Another good question is, is there any kind of tests that gives you good results without investing good-engineers time? If you find any, I'd ask you to share (but I would understand if you consi…

I have generally found that integration tests with well-mocked external dependencies achieve 80% of the things E2E tests do with a quarter of the effort.

Re: We killed our end-to-end test suite

#138

Post by 3pt14159 »

arp242 wrote: post #112
3pt14159 wrote: post #96

Earlier quoted context omitted.

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything. The way I've always viewed E2E tests is as "testing everything at the top…

Yes exactly. This mirrors my experience as well. Of course it depends on the individual setup, but making testing easier and faster has huge benefits, but a tiny amount of E2E coverage goes a long way.

Re: We killed our end-to-end test suite

#139

Post by cratermoon »

arp242 wrote: post #103
jamil7 wrote: post #94

Earlier quoted context omitted.

I’ve run into this a few times with some upstream package breaking and showing up in tests. I try to avoid mocking as much as possible in tests these days.

One thing I've done is adding the ability to run tests both with a "mocked" and a "real" version. The mocked version is fast and can be run quickly, the real version is much slower, but tests the actual real service. It's not that much extra effort to make in most cases, and I've caught some bugs when my mocked version made assumptions that were false, didn't cover some edge case, or whatnot. That said, I too avoid m…

I like the idea of testpoints in code that can be switched on or off, an idea originally from the hardware side. Modifying the testpoints to allow switching between different test implementations is a useful generalization of the idea.

Re: We killed our end-to-end test suite

#140

Post by catern »

lowbloodsugar wrote: post #110
daxfohl wrote: post #28

FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc. IMO these seem to be…

You have been in two of three situations. 1. No e2e tests. 2. Heavy, flaky, slow e2e tests. 3. e2e as a driver of first class system integration Compare with 1. no unit tests 2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes 3. CI/CD with 0 tolerance for failures or >1s tests The difference between "We have tons of tests" and "We driv…

>3. e2e as a driver of first class system integration

Could you say more about this approach? That's how I've tried to approach end-to-end tests, but I haven't found much written about this which is specifically about e2e tests.

Post reply on HN