Live data from Hacker News

PostgreSQL Scalability: Towards Millions TPS

akorotkov.github.io

71–80 of 222 posts

Re: PostgreSQL Scalability: Towards Millions TPS

#71
post #41
post #23

How does PostgreSQL compare to VoltDB? I'm trying to get a handle on the different databases, and VoltDB sounds exciting, but everyone's talking about PostgreSQL. Then there's Mnesia which I hear is, as all things Erlang, excellent, though it's kinda tied to Erlang. I know it's hard to say what's best, but what would you say is the best DB for a completely new multilingual project that needs throughput but prioritize…

WhatsApp I hear uses Mnesia. If you use Erlang and can fit everything in memory that does look pretty nice. It integrates right into the language.

It also came about before CAP was postulated, and does nothing to automatically resolve partitions. It doesn't -die-, but it was clearly built with a "we'll run this in our own data center" mentality. In the event of a partition, it goes immediately to split brain mode, which each side of the partition running separately, allowing both writes and reads, and doing nothing to try and automatically heal the partition, even if your Erlang nodes reconnect.

This isn't necessarily a bad thing; it's a very easy model to reason about, it rarely has an issue if your cluster doesn't span more than one data center, and it lets you know when it happens via events you can subscribe to. But doing anything different, up to and including healing the partition automatically, is left to the user.

There's also a few warts due to its history; disc only tables have a rather small max size (so if you're not expecting everything to be stored in RAM as well, you don't want to use Mnesia), indexes are shockingly inefficient for writes, and a few other odds and ends that I don't really remember. For persisted, but fast, storage in Erlang, where partitions aren't common, it's great, but outside of those sorts of use cases there's probably something better.

Re: PostgreSQL Scalability: Towards Millions TPS

#73
post #12

Slightly tangential but I'm genuinely curious, does any have a theory as to why nearly every RDBMS post on Hacker News is about Postgres and almost never MySQL or MariaDB? Considering the relative obscurity of the former it seems somewhat inexplicable.

Your perception of PostgreSQL being obscure in comparison to MySQL is interesting... Most projects I've worked on use pg. My perception is that projects using MySQL do so because of history (project started in php and that was the default stack, it was faster at the time (before transactions), etc...).

Re: PostgreSQL Scalability: Towards Millions TPS

#74

Are there any best practices for using PostgreSQL for storing time series data? Would it be comparable in performance to some of the NoSQL solutions (like Cassandra) for reasonable loads?

Do you mean timeseries data as in IoT/sensor data/etc, DevOps monitoring metrics (i.e. server load, app performance, etc), or something else?

Curious since I'm currently researching how PostgreSQL could do better in this space :)

Re: PostgreSQL Scalability: Towards Millions TPS

#76

Are there any best practices for using PostgreSQL for storing time series data? Would it be comparable in performance to some of the NoSQL solutions (like Cassandra) for reasonable loads?

I'm also in a similar position where I'd like to store approximately 560k records / user / year. My understanding is that Cassandra doesn't support some useful queries that would be useful when business logic is less clear (like group by)[1]. I'm leaning towards using PostgreSQL with a dedicated write DB until performance becomes an issue.

[1] http://stackoverflow.com/questions/17342176/max-distinct-and...

Re: PostgreSQL Scalability: Towards Millions TPS

#77

Andres is a great coworker to have at Citus Data, though I first ran into him on the mailing lists shortly after starting at Citus myself. I was tasked with figuring out "why do certain read-only workloads fail miserably under high concurrency?" I had never touched PostgreSQL before, nor any Linux performance tools, but I noticed that replacing certain buffer eviction locks with atomic implementations could drastical…

I wonder, does Intel's TSX/HLE help with these workloads? If it's read-only then I'd expect that it'd be able to elide a lot of the locking (assuming the Intel-designed heuristics do the job).

Re: PostgreSQL Scalability: Towards Millions TPS

#78
post #12

Slightly tangential but I'm genuinely curious, does any have a theory as to why nearly every RDBMS post on Hacker News is about Postgres and almost never MySQL or MariaDB? Considering the relative obscurity of the former it seems somewhat inexplicable.

MySQL is the PHP of relational databases. You can do most things with it, but you don't choose it when you have the option to choose something better.

Re: PostgreSQL Scalability: Towards Millions TPS

#79
post #34
post #28

Earlier quoted context omitted.

Err, no. Python probably isn't even in the top 5 of languages startups choose for their main stack.

Not the most sound method but I was demonstrating the relative popularity of programming languages to my SO the other day using find in the Who's Hiring thread for May and python was the most popular. Surprised me as well.

A ton more companies have some Python scripts than Python as a primary development language; this skews naive keyword matching assessments of job postings.

Re: PostgreSQL Scalability: Towards Millions TPS

#80
post #31
post #12

Slightly tangential but I'm genuinely curious, does any have a theory as to why nearly every RDBMS post on Hacker News is about Postgres and almost never MySQL or MariaDB? Considering the relative obscurity of the former it seems somewhat inexplicable.

There's been a large migration off of mysql to postgres simply because Oracle got the rights to mysql when they purchased Sun. That is what made me consider , ditching mysql. The final straw came when I stood up a mysql 5.6 instance to use as a data warehouse for about 5TB of data (15 billion rows). To my horror after spending a few weeks on this project I discovered that mysql only supported a small subset of the SQ…

> There's been a large migration off of mysql to postgres

No there hasn't:

http://db-engines.com/en/ranking

Which makes sense since they aren't really playing in the same space. The benefits of PostgreSQL are largely lost on typical MySQL use cases (light load, simple CRUD access patterns, limited use of JSON/BSON etc).

Post reply on HN