Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

261–270 of 270 posts

Re: We killed our end-to-end test suite

#261

Post by cratermoon »

aidenn0 wrote: post #176
cratermoon wrote: post #141

Earlier quoted context omitted.

At one place I consulted, the fte lead ignored flaky tests and attributed failures to the tests being wrong. A few months later... The code that was failing intermittently was found to be using floating point types for money. Yeah, I'm gonna wanna fix that.

Right if you have flaky tests there are 3 acceptable responses: 1. Fix the test 2. Fix the code that is being tested 3. Say "well we don't need this software to be reliable anyways so let just stop running tests" But many places seem to adopt hidden option #4 "Run the tests and ignore failures" A related issue is dialing the tunables for warnings up to 11 and then not reading any of the warnings. Once I saw a case wh…

Option #4 is just option #3 but keeping the costs of running tests you ignore.

You're right about excessive warnings, but then sometimes note. Running `gcc -Wall` used to be considered madness, and if you did it now on a codebase that has been around a while and not kept clean, you'd drown in messages. The key is to turn it on from the very start and fix things when there are 10 warnings instead of 1000.

This decay happens with test suites, too. One or two tests start to fail, and instead of fixing them, people ignore the failures. A bit later, it's five tests, then 10, and pretty soon the programmers see the tests as broken instead of looking at the failures that let things get to the point where there are so many failing tests.

Re: We killed our end-to-end test suite

#262

Post by amw-zero »

firedark60 wrote: post #258
mbrodersen wrote: post #216

Earlier quoted context omitted.

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

+1 on this comment. We had issues with poorly written selenium tests and after rewriting new tests with cypress and better test practices, the e2e test are reliable enough to be used as canary testing in new environments without false positive flaking. It ultimately comes down to how much you're willing to invest in writing good tests. If you're suffering from seriously flakey e2e tests results, more often than not i…

You hit the nail on the head, using your own words.

> selective retry on failures

This is why your test suite passes, not because of an avoidance of outdated tech or poor practices. You rerun your flaky tests until they pass. That’s bad engineering, and the definition of non-determinism.

Re: We killed our end-to-end test suite

#263

Post by joshribakoff »

catlifeonmars wrote: post #197
joshribakoff wrote: post #187

Earlier quoted context omitted.

You can try to monitor that an endpoint responds quickly but how do you monitor that it responds correctly? At the end of the day both tests and monitoring are forms of verification Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts I’d say there is a grey area between mon…

Generally, I think tests are used to validate changes to your service code (often as a gate to release it to production). Whereas monitoring is used to detect issues external to your code (often operated in production). Edit: That is to say, what distinguishes testing from monitoring isn’t content, but purpose.

Monitoring can catch issues in the code. For example if an event is dead lettered or the application crashes unexpectedly, it triggers an alert, which may make you aware of some edge case you forgot to test. Both tests and monitoring can encompass validating code is running correctly, some even run their tests against production at regular intervals as a form of monitoring, for example see “datadog synthetic tests”, which could be characterized as both a test and a monitor. Many companies opting not to do traditional e2e tests actually still have them, they’re just running them against production instead of blocking CI (with the rationale they will prioritize fast detection and mitigation rather than trying to prevent bugs from entering prod)

Re: We killed our end-to-end test suite

#264

Post by fndex »

continuational wrote: post #238
fndex wrote: post #106

Earlier quoted context omitted.

clojure.spec is not a type system. Still, my point is, just using a typed language won't remove the need for contracts, you would still need to roll up something like Nubank did even if it means using MyPy reflection features. For example, imagine you have two services that communicates through a message queue. Service A produces X as a string, but Service B consumes X as an integer. You can type that, both services…

Hm? Give the message a type. Service A or B would fail to compile, depending on whether the static type of X is a string or an integer.

AFAIK, kafka messages doesn't have "types", and even if they did, you would be relying on an external system, not your type system. If you are not convinced, test it by yourself, create two services and a kafka topic and produce a message from one service to another with different types on each service.

Re: We killed our end-to-end test suite

#265

Post by hinkley »

mbrodersen wrote: post #255
hinkley wrote: post #240

Earlier quoted context omitted.

You can’t test all error cases end to end. If you can you have shitty error handling. Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests. My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never ge…

Of course you can. Part of the end-to-end test is to setup the test scenarios you want to test (including limited HD space etc.)

You're going to spin up a vm with the wrong system time, and then advance it between two operations that take 200 ms on a live system?

Bullshit.

Re: We killed our end-to-end test suite

#266

Post by aidenn0 »

cratermoon wrote: post #261
aidenn0 wrote: post #176

Earlier quoted context omitted.

Right if you have flaky tests there are 3 acceptable responses: 1. Fix the test 2. Fix the code that is being tested 3. Say "well we don't need this software to be reliable anyways so let just stop running tests" But many places seem to adopt hidden option #4 "Run the tests and ignore failures" A related issue is dialing the tunables for warnings up to 11 and then not reading any of the warnings. Once I saw a case wh…

Option #4 is just option #3 but keeping the costs of running tests you ignore. You're right about excessive warnings, but then sometimes note. Running `gcc -Wall` used to be considered madness, and if you did it now on a codebase that has been around a while and not kept clean, you'd drown in messages. The key is to turn it on from the very start and fix things when there are 10 warnings instead of 1000. This decay h…

The fix for both situations is similar though; dial down the {warning strictness|number of tests run} until you get a clean {warnings|test-run} then enable them one by one in order of how easy they are to fix.

Re: We killed our end-to-end test suite

#267

Post by strokirk »

amw-zero wrote: post #259
mbrodersen wrote: post #254

Earlier quoted context omitted.

Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.

My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies. Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config. Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the f…

I've never had any issues with the E2E tests in my company either, and I'm unsure where the flakyness would even come from. Using Cypress with great success.

Re: We killed our end-to-end test suite

#268

Post by roguas »

Too wrote: post #231
strken wrote: post #203

Earlier quoted context omitted.

I've been thinking about the benefits of only writing E2E smoke tests which cover a small number of critical paths quickly. Seems like most of their problems came about because they wrote more tests than they needed, with higher coverage than was necessary.

If you are only allowed to have a single test in your project, it would have to be the E2E smoke. I've seen systems where master don't even start up for weeks but project is still proud to present unit tests are green with 100% coverage. One single E2E smoke outweighs all those tests, catching everything from faulty configuration, infrastructure, interface assumptions, integrations, libraries and code bugs. After thi…

this.. I have been doing that in products from hardware to ai models. It is effective and gives reasonable feedback.

I even gave a baiting tech talk at one of the companies about how useless unit test really are. Useless is understating it, they are often counter productive. Especially "junior" people, who try and grind the testing mindset, will come and write a little test for everything. Good luck making small inconsequential changes in the future.

If you have well written unit test (high level apis) then they become worthwhile - but once you arrive there, you have to just do one little step further to move into integration test and get good cover from failures/changes/bugs in cloud/infra provider.

Re: We killed our end-to-end test suite

#269

Post by mbrodersen »

hinkley wrote: post #265
mbrodersen wrote: post #255

Earlier quoted context omitted.

Of course you can. Part of the end-to-end test is to setup the test scenarios you want to test (including limited HD space etc.)

You're going to spin up a vm with the wrong system time, and then advance it between two operations that take 200 ms on a live system? Bullshit.

What a BS straw man answer. There are much easier ways to do that kind of testing.

Re: We killed our end-to-end test suite

#270

Post by dcow »

amw-zero wrote: post #244
dcow wrote: post #157

Earlier quoted context omitted.

Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.

This argument does not account for the fact that all e2e tests are non-deterministic, so the quality of your “setup” is not relevant.

You must have worked on some incredibly bad software to have non-determinism dominate your life. If a request fails then retry it just like a user would. If it keeps failing there's a problem. If it works then move on. I doubt your bank just throws in the towel and says "whelp software systems are inherently non-deterministic so we'll just forget some transactions here, allow the wrong amount over there, forget tests they're hard we can handle a little chance in our payment flows". The closest thing I've heard to that is amazon very occasionally shipping multiples of the same item because it was allegedly more expensive to implement immediate consistency than to ship a few duplicate items.
Post reply on HN