21–30 of 258 posts
How does PostgreSQL 10 compare with Cassandra for BigData requirements?
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
If anyone even remotely involved with the maintenance and development of pg reads this thread - Thank you! - for all your efforts in building and improving a first class product that keeps me amazed at the strides it takes with each major. release.
How does PostgreSQL 10 compare with Cassandra for BigData requirements?
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
Besides, if we picked machine size as _the_ definition, it would change every year. And that feels like a bad way to talk about big data.
(BTW, I'm not a huge fan of the term in general: I've seen a lot of companies sold a bill of good because some sales guy convinced them they needed a Big Data solution instead of something like Postgres)
Is this essentially a pipe dream?
How does PostgreSQL 10 compare with Cassandra for BigData requirements?
By no means an expert, but isn’t it Apples and Oranges? Cassandra is an eventual consistent database. Postgres is ACID. My 2c: you don’t need Cassandra or other BigData databases. If you did, you probably wouldn’t ask the question.
...and by the time you do, Postgres improvements will have rendered your Big Data medium-sized anyway :-)
[1]https://docs.microsoft.com/en-us/sql/relational-databases/ta...
Earlier quoted context omitted.
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
I've heard this 6TB number thrown around a bit recently, but no one can actually provide me with links to hardware I can buy - has anyone successfully setup a machine with this much ram who can post links to what they used?
Earlier quoted context omitted.
It's an improvement for sure, but I am curious -- does anyone situate a Postgres instance where it is publicly accessible? Who was asking for this feature?
Lots of cloud managed dbs are/can be publicly accessible. Eg all heroku hosted Postgres instances
How does PostgreSQL 10 compare with Cassandra for BigData requirements?
My team lead did a talk at NGINX Conf 2017 which touched on how we tuned Postgres to handle 1 Billion metrics per day for NGINX Amplify (https://www.nginx.com/products/nginx-amplify/). I would link it, but it seems it hasn't been published by our Conf team yet.
We did this on 9.4 so the changes outlined here regarding native partitioning, parallel queries, replication improvements, etc. would likely make doing a similar scaling structure easier and just straight up more performant.
I wouldn't classify 1 Billion metrics a day as "Big Data". I also would point out that our use case is unfairly biased to "recent" data rather than "all historical" data...which allows us to make some query optimizations that wouldn't be acceptable in the latter use case.
But we are using Postgres to host a constant, write-heavy workload that conventional wisdom dictates it is unsuited for. Moreover, it was/is fairly simple to implement, understand, and scale indefinitely. Personally, I think both Redshift and CitusData are various degrees of proof that scaling Postgres to handle these types of workloads is possible.
For me, the reason you want to consider using a Big Data focused tool (such as Cassandra) is at some point you end up fighting your tool to make a square peg fit in a round hole. At some point it makes more sense to adopt a specialized tool rather than work to specialize your implementation of a specific tool.
In all fairness, we are considering just that. Our road map actually has Scylla in the near future. It's not that we don't think Postgres could handle the load, but rather we think it will be easier to move specific data to Scylla and in the long run will save us resources (primarily in man hours and potentially in compute requirements as well).
Even if we move all our timeseries data to a specialized tool, we won't be getting rid of Postgres. For most/if not all our other persistence requirements Postgres is more than adequate. This is one of the reasons we chose to start with Postgres: even once we hit a scale that specialized tools made sense, Postgres would still be useful for everything else that doesn't require a specialized tool.
Moral of the story: Postgres will get you a lot farther than many give it credit for...and if you are just starting out I would highly encourage you to just pick Postgres and see how far it can take you. If you hit the limit, replace that specific part.
Edit: Also, there appears to be some movement on Columnar indexes which is worth keeping an eye on: