Thoughts on PostgreSQL in 2024

Thoughts on PostgreSQL in 2024
jkatz05.com

I have a huge DB which would take many TB but now runs on a 500GB footprint. Thanks to Rocksdb/MySQL. Pondering a move to Postgress, but seemed like a step down in this regard.

What could this be? Some crypto exchange is based on PostgreSQL?

I generally don't understand this kind of a requirement. Downtime, to me, is inability to have an end to end transaction. During downtime, to me, it is possible to queue writes and cache reads, to some extent.

I can understand if there is something in the middle, in the database layer, that needs to happen, but what would it be, that requires sub 1s latency, at that level?

Tickets are checked, by scanning a QR code, as people walk though the door. Not only do people hate sitting in their seat waiting (or worse, standing in line waiting)... you also need to pay a couple hundred employees to stand around and do nothing while waiting - ballpark cost of wages might be $5 for every second you can cut from the process of people walking through the door. On top of that wages cost, people waiting for the show to start don't buy drinks at the bar. A lot of events make more money on drink sales than ticket sales... often a high percentage of ticket sale revenue goes to whoever has a copyright claim to the show. They often get a cut of drink sales too, but it's a smaller cut.

So there are strong incentives to start scanning tickets as late as you possibly can before the performance actually starts. And if there's 1 second of downtime... it might trigger ten minutes of troubleshooting (turn it off and on again, etc) by a dozen people.

A show starting ten minutes late can be _really_ bad. If the show prep starts at 6pm and show pack up finishes at 11pm, that's five hours where some people are working non stop throughout that time - it's not really OK. Live events are dangerous, workplace deaths are far too common and some of the steps taken to try to prevent them don't really leave room for someone to take a meal break during a show. Alec Baldwin was shot and killed by a "prop" gun, in part because crew took a meal break at a critical time on the set, which interrupted routine safety checks... stuff like that can, and does, happen in theatre too.

There are union rules around breaks that everything is planned around and because delays happen in the real world there are financial penalties in place to discourage delays. So instead of $5 per second, that ten minutes might cost you $20 per second... in other words a $12,000 cost that wasn't in your budget... and the producer will have to explain why they ran $12k over budget - keep in mind the show might not actually make $12k profit in a single night's performance. Sometimes they don't even make a profit when they run smoothly.

All of that means your one second of downtime will be followed a formal written incident report and meetings to discuss what happened, and those meetings will inevitably include investigating alternative ticket services. We know little things like that can cost us a fortune, because we win new customers all the time when they tell us they left a competitor after a single brief period of downtime (our biggest competitor recently had a bug where buying tickets didn't work in the shitty browser that ships with certain cheap android phones... they fixed it relatively quickly but we gained a lot of new customers anyway).

Sure - it's only that critical from, say, 7:30pm to 7:55pm in a single building... an hour of downtime at 10am wouldn't be noticed... but if it's a global service, with dozens of cities in every timezone, then that critical state is happening 24/7/365.

One second of downtime will absolutely cause a measurable, and large, cost to your company profits. We deployed a major new system for ticket scanning in February last year. It worked perfectly, we haven't made any changes since February - other than just testing it out, and we're starting to deploy it over the next two months. That's how reluctant we are to break things.

We considered PostgreSQL, but decided it's not reliable enough.

We're using SQLite - with literally thousands of databases — every hand held scanning device as well as the servers run their own database. The servers themselves have a separate database for each individual event (and there are redundant servers too, so multiple server side databases per event). Also the database for critical things like scanning tickets is a separate database to less critical things, like payment records, event details, etc.

PostgreSQL is better than SQLite for the servers at least in a bunch of ways - but the big advantage of SQLite is flexibility - we can do a phased deployment where from the moment we start selling tickets (a few months before the actual event) to the moment of the actual event, there will be no deployments at all to that database unless we think a bug will directly effect the event. A deployment today will typically only apply to events that go on sale starting tomorrow.

Every business case has different needs, but skim reading this article on the future of Postgres, it sounds like they're working to improve some of the reasons I chose not to use it for this specific case.

[0] http://myrocks.io/