Live data from Hacker News

PostgreSQL Scalability: Towards Millions TPS

akorotkov.github.io

161–170 of 222 posts

Re: PostgreSQL Scalability: Towards Millions TPS

#161

Earlier quoted context omitted.

Honestly that's not something I would handle at the db layer. I would build a service that is responsible for keeping client db's in sync and abstract that away from my database entirely.

I'm not a fan of rolling our own sync code if I don't have to.

RethinkDB's new project Horizon could be a solution. But of course, that's not PostgreSQL.

Re: PostgreSQL Scalability: Towards Millions TPS

#162
post #81
post #3

Earlier quoted context omitted.

It's also a great example of a technology which is mature and well-tested yet actively growing and improving. Most open source projects have a lot to learn from Postgres.

What would you say those learning points are? What key factors would you introduce into another OSS project that PostgreSQL currently employs?

One of the weirdest is that there is no bug tracker. When things are found to be broken, they tend to get fixed immediately and there is nothing to track. Feature and incremental improvement work seems an arduous process, where you need to own your work and deal with extensive reviewing (no tossing it over the wall for others to maintain).

Re: PostgreSQL Scalability: Towards Millions TPS

#163
Postgres is still going through the motions of a transaction for every query you issue it even if nothing else but that transaction is happening on the server. So obviously if you add extra load in the form of writes, you may slow your reads down, but this was not a full benchmark, but instead a comparison of the same workload running against multiple versions of Postgres.

Re: PostgreSQL Scalability: Towards Millions TPS

#164

Man, how I wish WordPress had originally chosen to use PostgreSQL instead of MySQL back in the day.

Why? WordPress would not be any better or easier to use unless you already had PostgreSQL installed. Frankly for a use case as simple as WordPress you don't need to over optimize your database.

> WordPress would not be any better or easier to use unless you already had PostgreSQL installed.

Sounds like a situation I'm facing: assisting a non-profit org that wants a new website. Some members push for using Wordpress as the CMS. However the org has an established pgsql db containing lots of data that optimally could be employed on the site, e.g., lists of members/events, searchable documents, etc.

Also, there's a separate web app for accessing the database using SQL features mysql doesn't support. Lack of Wordpress/pgsql compatibility makes integrating the existing db and folding in the webapp functionality just about impossible, at least I'm not seeing how that could be done.

Since Wordpress isn't going to embrace pgsql, I've recommended considering alternative CMSs. Decisions are still pending.

Re: PostgreSQL Scalability: Towards Millions TPS

#165
post #124

Earlier quoted context omitted.

There is also Sequelize [1] which has more activity 1 - http://docs.sequelizejs.com/en/latest/

I haven't worked with Sequelize personally, but a friend has been recently and curses the day it was born - he wishes he'd used bookshelf.

Can you ask your friend about the exact details? From what I've seen it seemed OK.

Re: PostgreSQL Scalability: Towards Millions TPS

#166

Are there any best practices for using PostgreSQL for storing time series data? Would it be comparable in performance to some of the NoSQL solutions (like Cassandra) for reasonable loads?

I save 2 billion rows of timeseries data every year. I use a regular btree index for "hot data" that is less than 6 months old and BRIN index for older data. You can do this by writing a functional index.

You also have to spend some time to tune the query cost settings to avoid sequential scans if you're only gonna work with a subset of the data. Another optimization could be implementing table inheritance so you have a table for every year. If you work with data sets for a specific year you would get a big performance boost with sequential scans.

PostgreSQL's biggest weakness at the moment is aggregating data by using several cpu workers/cores. This is coming in PostgreSQL 9.6

Oh and I run PostgreSQL on ZFS with LZ4 compression,

Re: PostgreSQL Scalability: Towards Millions TPS

#167
post #133

Earlier quoted context omitted.

Because NoSQL is a hype. Many of the NoSQL essentially takes us back to 60s before Codd came up with relational model[1] These ideas tend to come back once in a while [2][3], but so far nothing is better than relational model. NoSQL still makes sense in many cases (generally when your specific use case does not need all guarantees of ACID), you can get in return higher performance or horizontal scalability. MongoDB i…

Currently MongoDB is outperformed by Postgres I hate hearing absolutist dogma like this. Some things are faster in Postgres, some things are faster in Mongo. We are migrating our analytics db from Mongo to PG but we hit a wall because SELECT COUNT(DISTINCT x)) performs abysmally in Postgres. Mongo's aggregation framework is an immature PITA but it performs this little trick well enough that we're pretty much stuck ke…

We are migrating our analytics db from Mongo to PG but we hit a wall because SELECT COUNT(DISTINCT x)) performs abysmally in Postgres.

An alternative is PipelineDB, which is built on Postgres (and drop-in compatible, supposedly) and should provide an efficient implementation for those kinds of queries: https://www.pipelinedb.com/

I've never used it, though.

Re: PostgreSQL Scalability: Towards Millions TPS

#168
post #28

Earlier quoted context omitted.

It's because Python tends to be the most common programming language for startups due to being easy to hire for, having good library support for a wide range of use cases, being good for web development, etc. And MySQL is focused on catering to enterprise customers who don't use Python. The last time I heard, they had something like one person working part time on Python support, so the drivers weren't nearly as reli…

Err, no. Python probably isn't even in the top 5 of languages startups choose for their main stack.

What is then?

One thing that's always bemused me is that YC doesn't insist on its companies using Lisp :-)

Re: PostgreSQL Scalability: Towards Millions TPS

#169

Earlier quoted context omitted.

Currently MongoDB is outperformed by Postgres I hate hearing absolutist dogma like this. Some things are faster in Postgres, some things are faster in Mongo. We are migrating our analytics db from Mongo to PG but we hit a wall because SELECT COUNT(DISTINCT x)) performs abysmally in Postgres. Mongo's aggregation framework is an immature PITA but it performs this little trick well enough that we're pretty much stuck ke…

There's no reason for SELECT COUNT(DISTINCT x)) to perform badly in Postgres, as long as you have an appropriate table design and indexes. Also MSSQL's query planner isn't better than Postgres', I work with both. Postgres does have its quirks though, especially with the MVCC row expiry.

You would need to add support for skip scans to the query planner, and adding feature to the query planner is rarely simple.

Re: PostgreSQL Scalability: Towards Millions TPS

#170

Earlier quoted context omitted.

Why not use PostGIS for geo-spatial data?

For processing, 100% agree. However if the write load is very high then Mongo is better suited as the intial store. I then replicate to other dbs.

Hm, fast bulk write is an area where PostgreSQL generally handily beats MongoDB. I do not see why it would be different for geodata. MongoDB beats PostgreSQL for some things, but not bulk load.
Post reply on HN