Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pivotal Greenplum Database has been open sourced

github.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Pivotal Greenplum Database has been open sourced

#21

Post by threeseed »

lobster_johnson wrote: post #16
nl wrote: post #15

Earlier quoted context omitted.

Greenplum is designed for datawarehousing. The analytics it is built for aren't the kind that ElasticSearch is typically used for. Greenplum/datawarehouse style analytical are things like: What is the average spend of female customers with 2 children who live in postcode AAAA or BBBB. Break it down by day of week and group by their marital status. Now for the top cohort of buyers on Mondays, give me a breakdown of th…

Thanks, that makes a lot of sense. We have those types of queries, too, which is where ES definitely breaks down. We wouldn't mind switching those parts (the "BI") of the analytics into something with higher latency.

One option there is to use Spark.

You can then write SQL, Scala, Python, R to interact with ElasticSearch. I can't recall the performance but against Cassandra, HDFS, HBase, MongoDB etc. it is very fast.

Re: Pivotal Greenplum Database has been open sourced

#22

Post by threeseed »

mavam wrote: post #20
lobster_johnson wrote: post #13

Earlier quoted context omitted.

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

Could you elaborate on the issues you have with ElasticSearch? Performance/Scaling? Usability?

I've attempted to load huge amounts of data into ElasticSearch. It is a bit fiddly to disable indexing and the sheer write performance is poor against dedicated databases. It is a testament to how good ElasticSearch is that people do use it as a database.

Re: Pivotal Greenplum Database has been open sourced

#23

Post by x0x0 »

lobster_johnson wrote: post #13
jacques_chester wrote: post #12

Earlier quoted context omitted.

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

I used greenplum until about three years ago when that employer replaced it. There is no way to describe it other than brittle and unsuited for production work. My peers discovered multiple ways to kill the db, all with data loss. Before you use it you should find a current user. It is one of a handful of techs on my personal "never again" list.

Licensing was also ludicrously expensive, including an attempt to multiply our license cost by 10 one year. I'd be very leery of lock-in given the owners. I also suspect this is the result of a failed attempt to sell it and is an open-source hail mary.

Re: Pivotal Greenplum Database has been open sourced

#24

Post by polskibus »

What's the story behind greenplum? Is it an old startup that has been bought by pivotal some time ago? I found an article about greenplum raising 20 mln $ in 2008 [1]

I was wondering whether anyone used greenplum in production and with what kind of loads. With today's in-memory fad, I am also interested in whether gp model supports loading everything into memory in a MOLAP fashion.

[1] http://techcrunch.com/2008/01/21/greenplum-takes-27-million-...

Re: Pivotal Greenplum Database has been open sourced

#25

Post by LittlePeter »

piggybox wrote: post #19

would like to see some benchmark vs Redshift, though the latter is a blackbox

What do you mean blackbox? It has explain analyze, it has rich query meta info, it has a web interface to query stats. You even know on what hardware it runs. Genuinely not sure what's blackbox about Redshift.

Re: Pivotal Greenplum Database has been open sourced

#26

Post by blumkvist »

nl wrote: post #15
lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

Greenplum is designed for datawarehousing. The analytics it is built for aren't the kind that ElasticSearch is typically used for. Greenplum/datawarehouse style analytical are things like: What is the average spend of female customers with 2 children who live in postcode AAAA or BBBB. Break it down by day of week and group by their marital status. Now for the top cohort of buyers on Mondays, give me a breakdown of th…

Hey, forgive my ignorance, I'm the guy who uses this stuff, not the guy who builds it.

My understanding was that, typically, the DW is where the data gets ETLed from various places and lives in a normalized environment. Then from that big pool, departments get built columnar or OLAP databases for specific analyses they want to do, and I connect not to the DW but to the OLAP instance (we call it Datamart here). Was I living a lie? At first I thought that your master-segment system is the same, but then I read your comment. I looked into Greenplum a while back and wanted it because of Madlib, but my guys said "nope" and went with MS.

Re: Pivotal Greenplum Database has been open sourced

#28

Post by eis »

x0x0 wrote: post #23
lobster_johnson wrote: post #13

Earlier quoted context omitted.

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

I used greenplum until about three years ago when that employer replaced it. There is no way to describe it other than brittle and unsuited for production work. My peers discovered multiple ways to kill the db, all with data loss. Before you use it you should find a current user. It is one of a handful of techs on my personal "never again" list. Licensing was also ludicrously expensive, including an attempt to multip…

I'm curious as to what the replacement was and if you were happy with it.

Re: Pivotal Greenplum Database has been open sourced

#29

Post by tlrobinson »

__david__ wrote: post #27

What's up with just a single commit since 2006? https://github.com/greenplum-db/gpdb/commits/master Why include all the really old commits, while squashing the most recent 10 years of commits into a single commit (6b0e52bead)?

The old source history is Postgres itself, the latest single commit is Greenplum.

I wonder why it's based on Postgres 8.2 rather than a newer version?

Post reply on HN