This site is blocked on my office proxy under "Adult/Mature Content" category
Testing Database Transactions in Go
11–20 of 22 posts
Re: Testing Database Transactions in Go
#12In order to get proper error response when deadlock occurs in testing (not just in Go), use isolation level serializable on every transaction / connection. This is simplest most effective way I have found so far, also you can get in detail why/where it happened. Watch out for different packages test executions since they run in parallel by default, use
go test -p 1
Re: Testing Database Transactions in Go
#13This problem is solved in most languages by wrapping endpoint code in an interceptor that creates and cleans up the transaction for you.
I'm not a fan of Go because it encourages hacks like this. The language is not expressive enough to handle database transactions properly. So it's suggested to try to catch bad situations with a linter instead.
What a timeless Goland solution to the problem. They should put "fix it with a linter" on hats and sell them at Go rallies. You could make billions
Re: Testing Database Transactions in Go
#14This site is blocked on my office proxy under "Adult/Mature Content" category
Yeah because of my last name :( Some filters seem to be a bit too aggressive. I copied it to Medium:
https://medium.com/@dekugelschieber/testing-database-transac...
Re: Testing Database Transactions in Go
#15Don’t forget to measure first. You can metric out the number of connections periodically and see if you are capping out. See https://golang.org/pkg/database/sql/#DBStats
Thanks! Thats a really good tip. I didn't know I had easy access to the db client stats!
Re: Testing Database Transactions in Go
#16This problem is solved in most languages by wrapping endpoint code in an interceptor that creates and cleans up the transaction for you. I'm not a fan of Go because it encourages hacks like this. The language is not expressive enough to handle database transactions properly. So it's suggested to try to catch bad situations with a linter instead. What a timeless Goland solution to the problem. They should put "fix it…
> an interceptor that creates and cleans up the transaction for you.
I'm not sure about the word "interceptor" here because in Go and every other language I know which does this it is merely a wrapper, but Go does this. From some vantage, the problem only arises because Go does this - if it instead forced you to create a transaction the resource allocation, and so possibility of exhaustion, would be a lot more obvious.
Re: Testing Database Transactions in Go
#17Earlier quoted context omitted.
Great points! Any hints regarding preexisting linters for go code that errors if we execute outside (an already opened) db transaction? Also, regarding writing tests against real PostgreSQL databases, isolated per test and without losing speed, I can recommend to take a look at https://github.com/allaboutapps/integresql - disclaimer, we are the authors.
> Great points! Any hints regarding preexisting linters for go code that errors if we execute outside (an already opened) db transaction? I'm limiting the open connections at the moment and using the default Goland linter. It won't show you these kind of errors though and I just remembered that I saw it somewhere else, but I don't remember where it was :D
You might be thinking of https://github.com/ryanrolds/sqlclosecheck but it will only catch leaks, not deadlocks.
Re: Testing Database Transactions in Go
#18This covers one possible deadlock, but there are so many more ways to do this. More common (IME) is a true logic deadlock, particularly if you are using strict serialization (say, your DB stores a ledger of financial transactions).
It’s actually really hard to test that your DB is using transactions correctly, since the error cases tend to be races which don’t show up under low load.
We built a prototype tool in Python where you can monkeypatch the transaction context manager, and pause the primary thread’s execution just before committing, so that you can then do evil stuff like running a competing thread of the same or other DB operations to try to break invariants. But even this won’t catch everything; there is a combinatorial explosion of test points and you can’t compare them all in a large app.
I haven’t seen any folks writing about this, another approach would be to wire up some him thing smarter like Jepsen to direct the anomaly search, Kyle said some folks have reported doing this but nobody has published. I’m interested to know if anyone has had success with this sort of transaction/correctness testing.
Re: Testing Database Transactions in Go
#19Python context managers are a great way to prevent this type of issues.
Anyway feels like a bit strange to not use the current connection (from between the transaction) to query things. Perhaps the problem is between the API, because it's not explicit the borrow/leave of connections from the pool.
Re: Testing Database Transactions in Go
#20This covers one possible deadlock, but there are so many more ways to do this. More common (IME) is a true logic deadlock, particularly if you are using strict serialization (say, your DB stores a ledger of financial transactions). It’s actually really hard to test that your DB is using transactions correctly, since the error cases tend to be races which don’t show up under low load. We built a prototype tool in Pyth…
Most people use queues to do this. At least I do.