Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

171–180 of 270 posts

Re: We killed our end-to-end test suite

#171

Post by lamontcg »

mabbo wrote: post #158

This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running suite So speed up your tests. Run them in parallel. Find better frameworks for running tests. > Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; Fix your flaky tests! Why anyone just acce…

> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra?

We actually do pretty well testing against fake infra.

We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightweight server written in ruby which would never scale but is the same API. When the test suite is updated both the implementations need to be fixed. When the client code runs integration tests we can test against the lightweight ruby code and when it passes we actually have pretty high confidence that it runs against the production code. We have hundreds of those tests and they can be run as fast as spinning up a ruby process with an in-memory datastore which is trashed on every test. Compare that to end to end tests that might fire up a set of images, terraform them into production servers and clients and run a scripted interaction or set of interactions and then throws that away and does it again.

At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. The right strategy is that you want to have enough of the most realistic tests to give you a high level of confidence that your faster, slightly less realistic tests are useful, and by having those faster tests you increase your amount of coverage, and on down the stack iteratively until you may get to unit tests of individual objects.

Re: We killed our end-to-end test suite

#173

Post by jart »

grumple wrote: post #156
mattnewton wrote: post #149

Earlier quoted context omitted.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are pr…

You can if it's BSD-licensed.

Re: We killed our end-to-end test suite

#175

Post by aidenn0 »

grumple wrote: post #156
mattnewton wrote: post #149

Earlier quoted context omitted.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are pr…

If you use other people's software and hardware, and those things don't perform the way your software assumes they perform, knowing that would be useful, right? There's always a limit to how much you want to handle, but if you are having a test fail even a large fraction of 1% of the time, then there's probably some underlying behavior that you should account for in production as well.

Re: We killed our end-to-end test suite

#176

Post by aidenn0 »

cratermoon wrote: post #141
aidenn0 wrote: post #61

Earlier quoted context omitted.

Was in a similar situation, and the VP of engineering banned the practice of rerunning failed tests, so flaky tests caused everybody pain. In less than 8 weeks the false positive rate dropped by about 3 orders of magnitude. There's a strong tendency to treat tests as a hurdle to get over rather than to treat them as first-class part of the development process.

At one place I consulted, the fte lead ignored flaky tests and attributed failures to the tests being wrong. A few months later... The code that was failing intermittently was found to be using floating point types for money. Yeah, I'm gonna wanna fix that.

Right if you have flaky tests there are 3 acceptable responses:

1. Fix the test

2. Fix the code that is being tested

3. Say "well we don't need this software to be reliable anyways so let just stop running tests"

But many places seem to adopt hidden option #4 "Run the tests and ignore failures"

A related issue is dialing the tunables for warnings up to 11 and then not reading any of the warnings. Once I saw a case where the build generated 1000s of warnings. Found a bug and said "this would be flagged as a warning even with relatively low warning settings" sure enough it was.

Obviously fixing warnings is good, but if they had just lowered the warning setting to be something reasonable, they would have had maybe 10 warnings total, one of which was a bug, which is a lot more useful than 1000s of warnings, at least one of which was a bug.

Re: We killed our end-to-end test suite

#177

Post by williamdclt »

jseban wrote: post #152
bcrosby95 wrote: post #68

Earlier quoted context omitted.

I have a friend that works on a team whose whole job is writing e2e tests. Before them the tests were slow, buggy, and couldn't be ran in parallel. Now they can be ran in parallel and there are few-to-no false positives. There's still challenges with this model (such as tracking changes on other teams, helping ensure that UIs are testable), but it seems to have worked out much better for their company than expecting…

Yeah but what's the point then, in that case you can just take back the old QA team and delete the gazillion lines of e2e test code and save yourself the liability of all that complexity. If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

> If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

Because you don't want to make _a_ manual test, you want to make _hundreds_ of tests.

Re: We killed our end-to-end test suite

#178

Post by mabbo »

lamontcg wrote: post #171
mabbo wrote: post #158

This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running suite So speed up your tests. Run them in parallel. Find better frameworks for running tests. > Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; Fix your flaky tests! Why anyone just acce…

> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra? We actually do pretty well testing against fake infra. We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightwe…

> At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do.

Yeah, you're not wrong. I'm just griping because of the previous list of complaints.

There are some benefits to running your tests against a mirror infra to reality. But they are limited. The bugs they catch that you won't catch running fake infra are very small in number, but terrifying in difficulty to solve.

Re: We killed our end-to-end test suite

#179

Post by gorgoiler »

The sad fact of E2E is that the tests genuinely find broken stuff. The “false negative” test results usually just mean false as in “something was broken, just not what the CI claimed was broken.”

It’s could be anything, so you need automatic specificity as to what’s broken (hard) or buy-in from the entire organisation to be on standby for finding broken stuff (also hard.)

“Anything” as in if your external DNS provider has 1 of 10 resolvers with an out of date zonefile, or a dodgy switch port to that particular resolver.

It’s hard but if it’s broken then it’s likely it is a real issue one of your end users is also experiencing. A commitment to E2E is committing to a level of quality across your entire infrastructure that few people are prepared to own.

Re: We killed our end-to-end test suite

#180

Post by williamdclt »

evandwight wrote: post #131
jdsleppy wrote: post #126

Earlier quoted context omitted.

As one example, Django handles test parallelism by creating N test databases (on the single test database server) and dividing tests into N runners. https://docs.djangoproject.com/en/3.2/ref/django-admin/#envv... You could also have multiple Docker containers running DBs.

Thank you! That's so crazy!

Not so crazy, it's very feasible to roll it yourself! Postgres has a "copy database" feature that's very useful (`CREATE DATABASE xxx WITH TEMPLATE yyy`).

I saw a project on HN a while ago focused on "managing isolated PostgreSQL databases for your integration tests", never used it but looks like a good idea: https://github.com/allaboutapps/integresql

Post reply on HN