Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Two Month Debugging Story

kev.inburke.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: A Two Month Debugging Story

#21

Post by jwatte »

What we do when we detect a failure is freeze the test runner instance, and allocate the same failed test to another runner. If the second runner succeeds, we okay the build, but we put the test and the frozen runner in a queue for investigation, and some engineer will be responsible for diagnosing and fixing this intermittent test. This queue is worked every day on a rotating schedule.

We run our own CI/CD infrastructure, on top of virtualized infrastructure; if you use a third party that doesn't give you that, you might want to look for alternatives.

Re: A Two Month Debugging Story

#22

Post by jwatte »

agf wrote: post #18
kevinburke wrote: post #11

Earlier quoted context omitted.

Hi, I wrote the post. We'd love to use transactions but our ORM doesn't support it. More info here: https://kev.inburke.com/kevin/dont-use-sails-or-waterline/ We wrote our own transaction library and have been shifting queries over to it/a saner ORM. https://github.com/shyp/pg-transactions . I've never heard of database templates, I'll have to take a look.

If your ORM doesn't know about transactions, that doesn't necessarily prevent you from wrapping your test runs in transactions. I was in a situation where I needed to start / rollback a transaction without ActiveRecord knowing about it, and was able to hack AR slightly to make that work -- so you can likely do the same. You even mention in the Waterline section of that blog post that you've written your own transacti…

Or run your database on a RAM disk with snapshots (LVM, FS, block device, or virtual host based)

Re: A Two Month Debugging Story

#23

Post by fapjacks »

They are using CircleCI, btw. I really like Circle, but they could sure be more helpful with the SSH access, as you mention. Also a personal nitpick, they (still) have no way to test a circle.yml file except for pushing a change and seeing if it works. Anybody that's used the service will know exactly what I'm talking about. You end up pushing four times just to get the syntax right. They know about that (iirc there was a support forum post requesting the feature some years ago), too. But besides that, I heartily recommend Circle.

Re: A Two Month Debugging Story

#24

Post by jwatte »

kevinburke wrote: post #15
gerbilly wrote: post #14

With Hibernate we used to connect it to a in memory DB just for the tests and to a real DB for production. It was easy and quick to drop schemas.

I agree that would be faster, but we rely a lot on Postgres's database constraints and we've previously had code problems go undiagnosed when the test schema didn't match production. Six in one, half a dozen in the other, I guess.

Why does your test infrastructure allow a schema that doesn't match production? Seems like low hanging fruit there!

Re: A Two Month Debugging Story

#25

Post by akiselev »

jwatte wrote: post #21

What we do when we detect a failure is freeze the test runner instance, and allocate the same failed test to another runner. If the second runner succeeds, we okay the build, but we put the test and the frozen runner in a queue for investigation, and some engineer will be responsible for diagnosing and fixing this intermittent test. This queue is worked every day on a rotating schedule. We run our own CI/CD infrastru…

Has this strategy worked perfectly so far? I would imagine a best two of three method would work better like those used for integrity in high radiation environments. However, I'm guessing code is significantly more deterministic and the danger of false positives (tests passing when they shouldn't) is much smaller.

Re: A Two Month Debugging Story

#26

Post by overgard »

At risk of invoking the wrath of test fanatics -- aren't tests supposed to save time? If your tests are both unreliable and so hard to debug that you haven't been able to fix them for two months, I'd think you'd be better off just turning them off and figuring out a better testing strategy.

Re: A Two Month Debugging Story

#27

Post by kevinburke »

overgard wrote: post #26

At risk of invoking the wrath of test fanatics -- aren't tests supposed to save time? If your tests are both unreliable and so hard to debug that you haven't been able to fix them for two months, I'd think you'd be better off just turning them off and figuring out a better testing strategy.

Hi - we definitely weren't looking at this full time for two months, just off and on as it failed various test runs, and I had time to look at it.

We have a large suite of tests that integrate with Postgres. Maybe one out of 50,000 database queries would fail. It was hard/impossible to predict which query would fail, since the problem wasn't with any individual query. By your logic, we should throw out our entire test suite. I'm not sure what we're supposed to replace it with.

Re: A Two Month Debugging Story

#28

Post by overgard »

kevinburke wrote: post #27
overgard wrote: post #26

At risk of invoking the wrath of test fanatics -- aren't tests supposed to save time? If your tests are both unreliable and so hard to debug that you haven't been able to fix them for two months, I'd think you'd be better off just turning them off and figuring out a better testing strategy.

Hi - we definitely weren't looking at this full time for two months, just off and on as it failed various test runs, and I had time to look at it. We have a large suite of tests that integrate with Postgres. Maybe one out of 50,000 database queries would fail. It was hard/impossible to predict which query would fail, since the problem wasn't with any individual query. By your logic, we should throw out our entire tes…

> By your logic, we should throw out our entire test suite.

That's why I figured I was invoking the wrath of the testing fanatics... that is where my logic leads. :-) I have no idea what your tests look like, maybe they're super useful, but I've worked at places in the past where there were thousands of tests that were frankly pretty useless, and were actually a net negative for a variety of reasons, but to point that out got you labeled as a "cowboy". (To be clear I'm not against automated testing, just automated testing done badly.) If you're running them in a context where you can't debug it, at the very least I would move it in house onto a machine where you can hook up a debugger.

Re: A Two Month Debugging Story

#29

Post by kevinburke »

jwatte wrote: post #24
kevinburke wrote: post #15

Earlier quoted context omitted.

I agree that would be faster, but we rely a lot on Postgres's database constraints and we've previously had code problems go undiagnosed when the test schema didn't match production. Six in one, half a dozen in the other, I guess.

Why does your test infrastructure allow a schema that doesn't match production? Seems like low hanging fruit there!

Sorry - that was a few months ago, we replaced it as soon as feasible. We had to upgrade a lot of tests to make it work with our prod schema
Post reply on HN