How to Build Your Distributed Database
citusdata.com
How to Build Your Distributed Database
1–10 of 24 posts
Re: How to Build Your Distributed Database
#2Let me ask you this - how would I use citusdb in such a way that it does not become a tax on my growth?
In other words, big data usually precedes big revenue, but most data products are priced per datum not per revenue.
So to put in indelicately, who can afford this and if they could, why would they? (after all those who could afford it, eg: bloomberg have strong reasons not to)
Re: How to Build Your Distributed Database
#3Let me ask you this - how would I use citusdb in such a way that it does not become a tax on my growth? In other words, big data usually precedes big revenue, but most data products are priced per datum not per revenue. So to put in indelicately, who can afford this and if they could, why would they? (after all those who could afford it, eg: bloomberg have strong reasons not to)
After a very cursory look, isn't the answer to download and use one of Citus' free, open source extensions to the free, open source postgres? (Depending on your use-case either pg-shard or cstore-fdw)
http://citusdata.com/citus-products/pg-shard http://citusdata.com/citus-products/cstore-fdw
Re: How to Build Your Distributed Database
#4Let me ask you this - how would I use citusdb in such a way that it does not become a tax on my growth? In other words, big data usually precedes big revenue, but most data products are priced per datum not per revenue. So to put in indelicately, who can afford this and if they could, why would they? (after all those who could afford it, eg: bloomberg have strong reasons not to)
After a very cursory look, isn't the answer to download and use one of Citus' free, open source extensions to the free, open source postgres? (Depending on your use-case either pg-shard or cstore-fdw) http://citusdata.com/citus-products/pg-shard http://citusdata.com/citus-products/cstore-fdw
They require the paid version to do distributed joins.
Re: How to Build Your Distributed Database
#5Let me ask you this - how would I use citusdb in such a way that it does not become a tax on my growth? In other words, big data usually precedes big revenue, but most data products are priced per datum not per revenue. So to put in indelicately, who can afford this and if they could, why would they? (after all those who could afford it, eg: bloomberg have strong reasons not to)
Umur from Citus here. Our goal is to make CitusDB an enabler for your growth by making scaling out simple for you and your dev, ops and analyst teams. If we've made it a tax instead and haven't saved you significant time, effort and complexity in the process, we're not doing our job.
At a practical level, we offer several ways to accomplish this: - We provide free, open-source extensions on standard PostgreSQL (pg_shard, cstore_fdw) - We provide a free community edition of CitusDB for added functionality (e.g. massively parallel analytic queries, distributed joins) - For enterprises, we provide a sitewide, unlimited license of CitusDB Enterprise. For smaller projects there, we provide support and a per-node license. - For start-ups, we provide a flat rate of CitusDB Enterprise irrespective of your data volume.
The right approach depends on the company and the use-case. Either way, and given the quick time-to-deployment, any of the approaches should end up as a major cost saver.
Re: How to Build Your Distributed Database
#6I'd be interested in what Datomic's approach looks like in comparison.
Re: How to Build Your Distributed Database
#7I'd be interested in what Datomic's approach looks like in comparison.
From what I gather, underneath Datomic is an event sourcing database, which is a model that already scales "for free".
Further optimization is the fact the query engine lies on the application, so if you have N application servers you have N CPUs available for querying - as opposed to overloading a master server or having to provision read slaves.
Re: How to Build Your Distributed Database
#8Can someone explain why one can't simply average the individual average results as the author wrote below:
"" No, we can't run averages on worker nodes, and then average those out. We need to have each worker node compute their sum(order_value) and count(order_value), and then sum(sum()) / sum(count()) on the coordinator node. ""?
Thank you.
Re: How to Build Your Distributed Database
#9Can someone explain why one can't simply average the individual average results as the author wrote below: "" No, we can't run averages on worker nodes, and then average those out. We need to have each worker node compute their sum(order_value) and count(order_value), and then sum(sum()) / sum(count()) on the coordinator node. ""? Thank you.
Re: How to Build Your Distributed Database
#10Can someone explain why one can't simply average the individual average results as the author wrote below: "" No, we can't run averages on worker nodes, and then average those out. We need to have each worker node compute their sum(order_value) and count(order_value), and then sum(sum()) / sum(count()) on the coordinator node. ""? Thank you.
because (2+3+4)/3 != (2+3)/2 + 4/1