Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pivotal Greenplum Database has been open sourced

github.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: Pivotal Greenplum Database has been open sourced

#11

Post by lobster_johnson »

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation.

The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master?

The other concern is that this isn't just "horizontally partitioned Postgres". It's forked from a very old version (8.2) of Postgres, and so doesn't have things like hot standby, streaming replication, the JSON datatype, GIN indexes (does it have GiST?), arrays, etc. It looks like it's optimized for parallel workloads, not for general use.

Anyone here with any experience with Greenplum who can perhaps speak about their work and the things that Greenplum is good at?

Re: Pivotal Greenplum Database has been open sourced

#12

Post by jacques_chester »

lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master?

Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of heroism and victory that typically starred gpfdist as the magic sword.

In terms of what it's good at: really big queries on really big data sets. Note in particular the integration with the Apache MADlib (incubating) machine learning toolkit[2], which was opensourced and donated to the ASF earlier this year.

[1] http://gpdb.docs.pivotal.io/4330/utility_guide/admin_utiliti...

[2] http://madlib.net/

Re: Pivotal Greenplum Database has been open sourced

#13

Post by lobster_johnson »

jacques_chester wrote: post #12
lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Thanks, useful.

I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions.

We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the data.

But ES has its share of problems, and somethings like Greenplum would be preferable, if it can handle the write pattern.

Re: Pivotal Greenplum Database has been open sourced

#14

Post by jacques_chester »

lobster_johnson wrote: post #13
jacques_chester wrote: post #12

Earlier quoted context omitted.

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

I'm not sure, to be honest. Hopefully some of our greenplum specialists spot this post and hop in.

Re: Pivotal Greenplum Database has been open sourced

#15

Post by nl »

lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

Greenplum is designed for datawarehousing.

The analytics it is built for aren't the kind that ElasticSearch is typically used for.

Greenplum/datawarehouse style analytical are things like: What is the average spend of female customers with 2 children who live in postcode AAAA or BBBB. Break it down by day of week and group by their marital status. Now for the top cohort of buyers on Mondays, give me a breakdown of the 3 most popular products and our profit margin on each one. This is often called "Business Analytics" (BI)

You could make ElasticSearch do that of course, but it wouldn't be much fun.

Given that context, the single master limitation turns out not be be a huge problem. Typically Greenplum is setup to load data from the OLAP-style online system, and the amount of data loaded is very predictable.

Same with things like hot standby and replication. In the Datawarehouse world it isn't uncommon for there to be nightly periods where a batch dataload occurs and/or nightly reporting is done, and the availability of external interfaces to the DB during that time maybe restricted.

Re: Pivotal Greenplum Database has been open sourced

#16

Post by lobster_johnson »

nl wrote: post #15
lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

Greenplum is designed for datawarehousing. The analytics it is built for aren't the kind that ElasticSearch is typically used for. Greenplum/datawarehouse style analytical are things like: What is the average spend of female customers with 2 children who live in postcode AAAA or BBBB. Break it down by day of week and group by their marital status. Now for the top cohort of buyers on Mondays, give me a breakdown of th…

Thanks, that makes a lot of sense. We have those types of queries, too, which is where ES definitely breaks down. We wouldn't mind switching those parts (the "BI") of the analytics into something with higher latency.

Re: Pivotal Greenplum Database has been open sourced

#17

Post by parasubvert »

lobster_johnson wrote: post #13
jacques_chester wrote: post #12

Earlier quoted context omitted.

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

Gpfdist is really just a binary format over HTTP, so it does handle streaming use cases well.

Spring XD implemented a Java reactive version of the GPFdist protocol, so you could use that to wire it up to your existing app pretty easily (or just copy the Sink code which is pretty straightforward).

Here are some docs and synthetic benchmarks: http://docs.spring.io/spring-xd/docs/current/reference/html/...

Re: Pivotal Greenplum Database has been open sourced

#18

Post by parasubvert »

It's been a long time coming for this database niche to reach open source, which is the tech behind the big analytical prowess of countless companies (before and after Hadoop). Teradata, the first massively parallel database, was released in 1984.

A brief overview on MPP databases / Greenplum here: https://dwarehouse.wordpress.com/2012/12/28/introduction-to-...

Re: Pivotal Greenplum Database has been open sourced

#20

Post by mavam »

lobster_johnson wrote: post #13
jacques_chester wrote: post #12

Earlier quoted context omitted.

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

Could you elaborate on the issues you have with ElasticSearch? Performance/Scaling? Usability?
Post reply on HN