Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Citus Unforks from PostgreSQL, Goes Open Source

citusdata.com

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Citus Unforks from PostgreSQL, Goes Open Source

#111

Post by lfittl »

flavor8 wrote: post #95
umur wrote: post #87

Earlier quoted context omitted.

(Part 2) Now comes the storage engine, and cstore_fdw as it relates to PostgreSQL. Built by the Citus Data team, cstore_fdw is entirely a separate component from the Citus product above. It enables columnar storage for your vanilla, single-node PostgreSQL to provide data compression for faster analytics. As such, cstore_fdw does not come with any of the parallelism I've described above that Citus (or Redshift, Vertic…

Thanks for the answers - makes a lot of sense. I'm replying to this one just to clarify one point: > cstore_fdw has certain limitations, importantly it is not updatable Does it support insert, or are tables completely immutable once created?

Copying from https://github.com/citusdata/cstore_fdw#usage

  To load or append data into a cstore table, you have two options:
  
  You can use the COPY command to load or append data from a file, a program, or STDIN.
  
  You can use the INSERT INTO cstore_table SELECT ... syntax to load or append data from another table.
  
  Note: We currently don't support updating table using DELETE, and UPDATE commands. We also don't support single row inserts.
So I think you can certainly mutate tables, but the focus is on bulk-inserts, rather than individual append actions.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#112

Post by lfittl »

satygeek wrote: post #108

Does CitusDb fit in olap analytical workloads to do aggregations on hundreds millions of records using varying order and size of dimensions (eg druid) in max of 3 seconds response time using as few boxes as possible - Or there are other techniques have to be used along with Citusdb? Can you shed a light on your experience with CloudFlare in terms of cluster size and queries perf?

I'll leave the initial questions to the Citus team, but re: CloudFlare this link might be helpful:

https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...

Re: Citus Unforks from PostgreSQL, Goes Open Source

#113

Post by satygeek »

lfittl wrote: post #112
satygeek wrote: post #108

Does CitusDb fit in olap analytical workloads to do aggregations on hundreds millions of records using varying order and size of dimensions (eg druid) in max of 3 seconds response time using as few boxes as possible - Or there are other techniques have to be used along with Citusdb? Can you shed a light on your experience with CloudFlare in terms of cluster size and queries perf?

I'll leave the initial questions to the Citus team, but re: CloudFlare this link might be helpful: https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...

Thanks. I went through it but couldnt find info about their cluster size, data size and queries response time

Re: Citus Unforks from PostgreSQL, Goes Open Source

#114

Post by anarazel »

Someone wrote: post #105

One must thank them for open sourcing this, and cannot blame them for using a different license, but using a different license makes me think calling this "unfork" is bending the truth a little bit.

The "unfork" part is primarily about not forking the postgres codebase anymore, as done before citus 5.0 (i.e. we modified parts of postgres, to make it citus). Citus now entirely works as an extension to postgres, using the extension facilities postgres provides.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#115

Post by cm3 »

davidfetter wrote: post #73
gtrubetskoy wrote: post #18

Earlier quoted context omitted.

Which means there is no chance this would ever become part of PostgreSQL proper.

Citus owns the copyrights, and the CLA ensures that they will continue to. That means that they can reassign it under a different license if they so choose, and they could at some point choose a liberal license.

And a CLA in place also means many contributors will shy away from doing so because they have no idea what happens with their contribution in the next version. Either it's a community project owned by everyone under an agreed upon license or it's more like a record deal for new artists.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#116

Post by mslot »

Florin_Andrei wrote: post #102
ozgune wrote: post #22

Earlier quoted context omitted.

(Ozgun from Citus Data) We're working on making Citus masterless. In all openness, we evaluated two different approaches to this in the past six months, and wrapped up the design for one. This design works well on the cloud, and we already demonstrated a working version: https://youtu.be/_nun2S6EdWo?t=411 For on-premise deployments, the primary challenge is set-up complexity. We're now prototyping one of those design…

Would it be possible (eventually) to use Citus for sharding within the datacenter, and BDR for master/master replication between datacenters? Or is Citus taking over the master/master replication? (or is it doing something different?)

(Marco from Citus Data)

It seems at least theoretically possible.

Since the Citus master executes distributed queries by sending regular SQL queries to the Citus workers, you could already use BDR servers as workers and replicate the data between pairs of workers in different data centers and copy over the metadata on the master manually. However, some distributed joins and data loading features wouldn't work.

For all features to work, and to replicate the master, you would have to compile Citus against BDR, which probably requires a few code changes.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#117

Post by mslot »

buremba wrote: post #106
umur wrote: post #88

Earlier quoted context omitted.

(Part 3/3 - please see two comments below as the starting point) Aside from the different use-cases they address, there is one other, important difference between Citus and Redshift (and any other distributed database in the world, for that matter). Citus does not fork the underlying database, PostgreSQL. Instead, Citus extends PostgreSQL to transform it into a parallel processing, distributed database. We use Postgr…

I think that it would be better for you to position CitusDB by comparing it to other products in terms of use cases. If the data is big and I need to run analytic queries then I think I have to use a columnar storage format because row-oriented formats cause too much overhead for aggregation queries that usually need to process single column efficiently. If I use CitusDB as an analytical database, then it's comparabl…

(Marco from Citus Data)

> If I use CitusDB as an analytical database, then it's comparable with Redshift, Hive etc.

A particular difference is in response times and concurrency. Data warehouses and Hive are great for reporting use-cases, but not for use-cases that require fast responses and have many users like analytical dashboards. This is a use-case for which Citus is particularly well-suited (see for example the CloudFlare dashboard).

> Can I use cstore_fdw in CitusDB and able to take advantage of real-time nature of Postgresql?

Yes, since cstore_fdw and Citus are both developed by Citus Data we made sure they're fully integrated. We've even seen some deployments that use a mixture of columnar- and row-based storage in a single distributed table.

We find that row-based storage generally has better ingestion performance and more indexing possibilities. Citus can do very fast execution of analytical queries by parallelizing over row-based shards and using the indexes on each of them. However, if you only need a small number of columns and have analytical queries that are not very selective, you can use columnar storage just as easily and even mix and match (might require some support).

> I guess another use case is using CitusDB as distributed data store

Yep, Citus can definitely be used for that by using hash-partitioned tables.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#118

Post by Florin_Andrei »

mslot wrote: post #116
Florin_Andrei wrote: post #102

Earlier quoted context omitted.

Would it be possible (eventually) to use Citus for sharding within the datacenter, and BDR for master/master replication between datacenters? Or is Citus taking over the master/master replication? (or is it doing something different?)

(Marco from Citus Data) It seems at least theoretically possible. Since the Citus master executes distributed queries by sending regular SQL queries to the Citus workers, you could already use BDR servers as workers and replicate the data between pairs of workers in different data centers and copy over the metadata on the master manually. However, some distributed joins and data loading features wouldn't work. For al…

Postgres with sharding and master/master replication would be so awesome.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#119

Post by takeda »

Someone wrote: post #105

One must thank them for open sourcing this, and cannot blame them for using a different license, but using a different license makes me think calling this "unfork" is bending the truth a little bit.

Perhaps I'm missing something, but this is just an extension that works with standard postgres, there are no code changes in postgres itself, so it doesn't look like it ever was a fork.

Re: Citus Unforks from PostgreSQL, Goes Open Source

#120

Post by jasonmp85 »

takeda wrote: post #119
Someone wrote: post #105

One must thank them for open sourcing this, and cannot blame them for using a different license, but using a different license makes me think calling this "unfork" is bending the truth a little bit.

Perhaps I'm missing something, but this is just an extension that works with standard postgres, there are no code changes in postgres itself, so it doesn't look like it ever was a fork.

(Jason from Citus here)

Yes, that's what you're seeing right now, but in the past Citus (used to be "CitusDB") was a superset of the entire PostgreSQL codebase. During the lead-up to the open source release, we removed the use of any static methods or internal machinery and rewrote the installation process to use the PostgreSQL CREATE EXTENSION command. Additionally, we moved all of pg_shard's DML functionality into Citus to unify the product line.

So ultimately CitusDB was a fork but is now entirely an extension.

Post reply on HN