Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL Scalability: Towards Millions TPS

akorotkov.github.io

Post reply on HN
Print view View original

41–50 of 222 posts

Re: PostgreSQL Scalability: Towards Millions TPS

#41

Post by rdtsc »

tormeh wrote: post #23

How does PostgreSQL compare to VoltDB? I'm trying to get a handle on the different databases, and VoltDB sounds exciting, but everyone's talking about PostgreSQL. Then there's Mnesia which I hear is, as all things Erlang, excellent, though it's kinda tied to Erlang. I know it's hard to say what's best, but what would you say is the best DB for a completely new multilingual project that needs throughput but prioritize…

WhatsApp I hear uses Mnesia. If you use Erlang and can fit everything in memory that does look pretty nice. It integrates right into the language.

Re: PostgreSQL Scalability: Towards Millions TPS

#42

Post by ddorian43 »

tormeh wrote: post #23

How does PostgreSQL compare to VoltDB? I'm trying to get a handle on the different databases, and VoltDB sounds exciting, but everyone's talking about PostgreSQL. Then there's Mnesia which I hear is, as all things Erlang, excellent, though it's kinda tied to Erlang. I know it's hard to say what's best, but what would you say is the best DB for a completely new multilingual project that needs throughput but prioritize…

Mongodb is also in agpl. But the drivers probably aren't. So you can use it and be fine. And if you make changes to VoltDb you have to share them.

11.5 How much data can be stored in Mnesia?

Dets uses 32 bit integers for file offsets, so the largest possible mnesia table (for now) is 4Gb.

Re: PostgreSQL Scalability: Towards Millions TPS

#43

Post by qaq »

jlgaddis wrote: post #37
cwbrandsma wrote: post #29

Earlier quoted context omitted.

MySQL is considered more "old school" and not great at being standards compliant when installed with default settings. See https://twitter.com/robconery/status/189086889486192640 Also, MySQL is run by Oracle...and Oracle a horrible company run by horrible people that behave horribly, and should never ever be trusted in any form. Any developer relying or Oracle code should not be trusted ever again, and Oracle consult…

I am in no way a fan of Oracle but, c'mon, that's a little over the top, don't you think?

Reminds me of a talk by Bryan Cantrill he talked a bit (I am paraphrasing) that he didn't believe that people could be purely evil and when looking into Larry Ellisson he actually discovered he donated several hundred mil. to a nonprofit (dedicated to prolonging human life). On closer look it turned the sole purpose of that nonprofit was working on prolonging life of one particular human namely Larry Ellisson. (while writing off the donation from taxes to boot :).

Re: PostgreSQL Scalability: Towards Millions TPS

#44

Post by orf »

esaym wrote: post #31
jrcii wrote: post #12

Slightly tangential but I'm genuinely curious, does any have a theory as to why nearly every RDBMS post on Hacker News is about Postgres and almost never MySQL or MariaDB? Considering the relative obscurity of the former it seems somewhat inexplicable.

There's been a large migration off of mysql to postgres simply because Oracle got the rights to mysql when they purchased Sun. That is what made me consider , ditching mysql. The final straw came when I stood up a mysql 5.6 instance to use as a data warehouse for about 5TB of data (15 billion rows). To my horror after spending a few weeks on this project I discovered that mysql only supported a small subset of the SQ…

> after spending a few weeks on this project I discovered that mysql only supported a small subset of SQL

Did you not research this beforehand?

Re: PostgreSQL Scalability: Towards Millions TPS

#45

Post by matthewmacleod »

tormeh wrote: post #23

How does PostgreSQL compare to VoltDB? I'm trying to get a handle on the different databases, and VoltDB sounds exciting, but everyone's talking about PostgreSQL. Then there's Mnesia which I hear is, as all things Erlang, excellent, though it's kinda tied to Erlang. I know it's hard to say what's best, but what would you say is the best DB for a completely new multilingual project that needs throughput but prioritize…

You almost certainly want Postgres, unless you have a compelling, specific use case to use a database optimised for a specific workload, or for some reason Postgres isn't usable in your specific environment. It's a reliable, well-designed general-purpose RDBMS which will scale up pretty well to cope with fairly large workloads; performance and latency will not be problems with a sensible schema.

VoltDB, as an example, is very different: it seems to be designed for simple OLTP workloads. It's an in-memory database, which offers opportunities for impressive performance, but if you have a large amount of data, you'll need a large amount of memory. And horizontal scaling is cool, but cross-partition operations will incur significant overhead

(IANAL, but the AGPL requires network users of software be able to download the source. Since a presumably proprietary application is the client in this case, this isn't likely to be an issue.)

Re: PostgreSQL Scalability: Towards Millions TPS

#46

Post by anarazel »

Jweb_Guru wrote: post #38
anarazel wrote: post #30

Earlier quoted context omitted.

That's what the discussed padding patch does (except to only padding to 64bytes on most platforms). There's a downside though - on low concurrency the padding reduces the cache hit ratio sufficiently enough to cause a slowdown. Given we're in code freeze anyway, I've not spent a lot of though on that yet; but I suspect that rearchitecting things so the lines are dirtied fewer times, is the better fix; with less poten…

Would it be possible to dynamically choose the padding at server start time? Given that cache line sizes vary that seems like it might be a prudent choice.

> Would it be possible to dynamically choose the padding at server start time?

I doubt it. Allowing the compiler to generate accesses with lea et al. is quite beneficial; and that'd likely be gone by making this not be a compile time constant.

It'd also end up being a tuning knob very very few knew how to tune...

> Given that cache line sizes vary that seems like it might be a prudent choice.

They usually only vary between architectures. Netburst IIRC was the last time x86 cache line sizes varied. If there's any doubt it's usually ok to just use the higher (128 byte) line size, the "unused" padding cache-line will never be accessed and thus not occupy cache space.

Re: PostgreSQL Scalability: Towards Millions TPS

#47

Post by esaym »

orf wrote: post #44
esaym wrote: post #31

Earlier quoted context omitted.

There's been a large migration off of mysql to postgres simply because Oracle got the rights to mysql when they purchased Sun. That is what made me consider , ditching mysql. The final straw came when I stood up a mysql 5.6 instance to use as a data warehouse for about 5TB of data (15 billion rows). To my horror after spending a few weeks on this project I discovered that mysql only supported a small subset of the SQ…

> after spending a few weeks on this project I discovered that mysql only supported a small subset of SQL Did you not research this beforehand?

At the time, my basic experience with databases had been for normal developer CRUD type operations. I was just getting into BI analytics and stuff. MySql was kind of the defacto choice for a "free" database and it was kind of thrown at me. I didn't mind since I had used it for years and was familiar with setting it up. But no, assuming the thing with "SQL" in its name didn't actually support SQL was not on my checklist sadly.

Re: PostgreSQL Scalability: Towards Millions TPS

#48

Post by dimfeld »

devy wrote: post #13

In partnership with IBM we researched PostgreSQL scalability on modern Power8 servers. That statement and the linked Russian blog white paper[1] makes it seem like a Power8 specific and Power8 is a "a massively multithreaded chip"[2]. I wonder how far off it would be to x86-64? [1] https://habrahabr.ru/company/postgrespro/blog/270827/ [2] https://en.wikipedia.org/wiki/POWER8

The article says a few lines down, "The optimization #1 appears to give huge benefit on big Intel servers as well, while optimization #2 is Power-specific. After long rounds of optimization, cleaning and testing #1 was finally committed by Andres Freund."

Also, the chart indicates that the benchmarks shown were run on Xeon chips.

Re: PostgreSQL Scalability: Towards Millions TPS

#49

Post by the-dude »

qaq wrote: post #43
jlgaddis wrote: post #37

Earlier quoted context omitted.

I am in no way a fan of Oracle but, c'mon, that's a little over the top, don't you think?

Reminds me of a talk by Bryan Cantrill he talked a bit (I am paraphrasing) that he didn't believe that people could be purely evil and when looking into Larry Ellisson he actually discovered he donated several hundred mil. to a nonprofit (dedicated to prolonging human life). On closer look it turned the sole purpose of that nonprofit was working on prolonging life of one particular human namely Larry Ellisson. (while…

One Rich Asshole Called Larry Ellison

Re: PostgreSQL Scalability: Towards Millions TPS

#50

Post by twic »

jrcii wrote: post #12

Slightly tangential but I'm genuinely curious, does any have a theory as to why nearly every RDBMS post on Hacker News is about Postgres and almost never MySQL or MariaDB? Considering the relative obscurity of the former it seems somewhat inexplicable.

Are the MySQL or MariaDB communities publishing interesting stuff like this? That's a genuine question - i'm no longer a MySQL user, so i don't pay any attention to what's going on myself.
Post reply on HN