Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Two Month Debugging Story

kev.inburke.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: A Two Month Debugging Story

#31

Post by meshko »

jwatte wrote: post #22
agf wrote: post #18

Earlier quoted context omitted.

If your ORM doesn't know about transactions, that doesn't necessarily prevent you from wrapping your test runs in transactions. I was in a situation where I needed to start / rollback a transaction without ActiveRecord knowing about it, and was able to hack AR slightly to make that work -- so you can likely do the same. You even mention in the Waterline section of that blog post that you've written your own transacti…

Or run your database on a RAM disk with snapshots (LVM, FS, block device, or virtual host based)

I would imagine resetting the whole db even on a RAM disk after each of 6000 tests wouldn't be particularly fast.

Re: A Two Month Debugging Story

#32

Post by joshribakoff »

I'm not saying that e2e testing is without value, it can be useful... but its just not worth it I've found. I write unit tests only, which don't touch the database. My litmus test is am I testing an "algorithm" -- something that has outputs solely based on its inputs.

If you insist on tests that touch the DB, the speed can be improved by chaining tests instead of resetting the DB everytime (this has its trade offs).

If I need to test a piece of code that generates a dynamic SQL statement, I would simply assert that the correct SQL is generated. I would not need to actually execute the SQL to see that it is correct. The point is to test that my logic generated the correct SQL, not to test that my database vendor implements SQL correctly. The latter would just be caught in manual testing. I like the BDD school of thought that you are writing specs, not tests.

Re: A Two Month Debugging Story

#33

Post by jdmichal »

voiper1 wrote: post #3

First thought that came to mind was to just blow away the database and create a new one, see how long that takes. I had found waterline's load time to have a LOT of things that I hadn't expected and take a long time... I'm just using knex now. Every time I try for magical solutions, I keep regretting it and end up using less magic.

Magic is fine, but there should always be a path to do things sans pixie dust. For something like an ORM, that would be a path to retrieve the raw DB connection to do things around / between queries. You even see the same pattern play out in programming languages; C# has the `dynamic` keyword that disables type checking on a variable.

Re: A Two Month Debugging Story

#34

Post by primitivesuave »

joshribakoff wrote: post #32

I'm not saying that e2e testing is without value, it can be useful... but its just not worth it I've found. I write unit tests only, which don't touch the database. My litmus test is am I testing an "algorithm" -- something that has outputs solely based on its inputs. If you insist on tests that touch the DB, the speed can be improved by chaining tests instead of resetting the DB everytime (this has its trade offs).…

I second this. I worked at a research company that had terabyte-scale SQL databases of curated data, and the tests would take 5 - 8 hours to run because they were verifying query results rather than the queries themselves. This also meant tests would fail because of changes in the data rather than just changes in the code. I rewrote the entire test suite to match SQL statements against a symbolic SQL description, which turned out to be a much more robust and extensible solution.

Re: A Two Month Debugging Story

#35

Post by kevinburke »

overgard wrote: post #28
kevinburke wrote: post #27

Earlier quoted context omitted.

Hi - we definitely weren't looking at this full time for two months, just off and on as it failed various test runs, and I had time to look at it. We have a large suite of tests that integrate with Postgres. Maybe one out of 50,000 database queries would fail. It was hard/impossible to predict which query would fail, since the problem wasn't with any individual query. By your logic, we should throw out our entire tes…

> By your logic, we should throw out our entire test suite. That's why I figured I was invoking the wrath of the testing fanatics... that is where my logic leads. :-) I have no idea what your tests look like, maybe they're super useful, but I've worked at places in the past where there were thousands of tests that were frankly pretty useless, and were actually a net negative for a variety of reasons, but to point tha…

Our tests are pretty useful, they check things like "when you ask the API to submit a pickup, it gets assigned to a driver", and so on. Especially in JavaScript where anything can take any value they've been extremely helpful.

Re: A Two Month Debugging Story

#36

Post by meshko »

karmakaze wrote: post #7

> We could draw a dependency graph between our 60 tables and issue the DELETEs in the correct order If the above works, automate it.

creating a dependency graph between tables is not exactly a 20 minute task. Certainly doable, but getting it right would take time. Anyways, no need to do it, can just do the deletes in a transaction.

Re: A Two Month Debugging Story

#37

Post by mattmanser »

kevinburke wrote: post #35
overgard wrote: post #28

Earlier quoted context omitted.

> By your logic, we should throw out our entire test suite. That's why I figured I was invoking the wrath of the testing fanatics... that is where my logic leads. :-) I have no idea what your tests look like, maybe they're super useful, but I've worked at places in the past where there were thousands of tests that were frankly pretty useless, and were actually a net negative for a variety of reasons, but to point tha…

Our tests are pretty useful, they check things like "when you ask the API to submit a pickup, it gets assigned to a driver", and so on. Especially in JavaScript where anything can take any value they've been extremely helpful.

That's another argument I've never understood.

I've been working on applications with javascript for over 10 years.

Not once can I remember a javascript bug where someone passed a value as the wrong argument and it made it into production.

Maybe it has happened, but it's just not a common bug. I just don't remember it.

Like, you can easily write it as a mistake as you're developing but it's obvious as soon as you run it. But if that bug gets in production your dev didn't even bother to run the code to see if it worked as intended.

Re: A Two Month Debugging Story

#38

Post by polskibus »

kevinburke wrote: post #19
polskibus wrote: post #17

Have you thought about starting up multiple replicas of your database in parallel and then pointing each test suite to a different one? This way you may be able to parallelize them very well.

This is an interesting idea!

Let me know how it works out for you.

Re: A Two Month Debugging Story

#39

Post by rraval »

kevinburke wrote: post #11
rraval wrote: post #8

> To ensure each test has a clean slate, we clear the database between each test. The proposed solution to manually nuke the database state seems crazy to me. Some alternatives: 1. Run the entire test in a transaction, do flushes and assert as normal. At the end, ROLLBACK instead of COMMIT and now you have a pristine database again. [1] 2. Setup pristine DB state once and then use `CREATE DATABASE ... WITH TEMPLATE .…

Hi, I wrote the post. We'd love to use transactions but our ORM doesn't support it. More info here: https://kev.inburke.com/kevin/dont-use-sails-or-waterline/ We wrote our own transaction library and have been shifting queries over to it/a saner ORM. https://github.com/shyp/pg-transactions . I've never heard of database templates, I'll have to take a look.

One advantage of the template approach is it allows for test level parallelism, which may address your primary goal of performance.

> Waterline queries are case insensitive; that is, Users.find().where(name: 'FOO') will turn into SELECT * FROM users WHERE name = LOWER('FOO');. There's no way to turn this off. If you ask Sails to generate an index for you, it will place the index on the uppercased column name, so your queries will miss it. If you generate the index yourself, you pretty much have to use the lowercased column value & force every other query against the database to use that as well.

What the actual fuck!

Re: A Two Month Debugging Story

#40

Post by jwatte »

meshko wrote: post #31
jwatte wrote: post #22

Earlier quoted context omitted.

Or run your database on a RAM disk with snapshots (LVM, FS, block device, or virtual host based)

I would imagine resetting the whole db even on a RAM disk after each of 6000 tests wouldn't be particularly fast.

With snapshots, it should be instant.
Post reply on HN