Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pivotal Greenplum Database has been open sourced

github.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Pivotal Greenplum Database has been open sourced

#31

Post by petepete »

tlrobinson wrote: post #29
__david__ wrote: post #27

What's up with just a single commit since 2006? https://github.com/greenplum-db/gpdb/commits/master Why include all the really old commits, while squashing the most recent 10 years of commits into a single commit (6b0e52bead)?

The old source history is Postgres itself, the latest single commit is Greenplum. I wonder why it's based on Postgres 8.2 rather than a newer version?

That's probably why it was open-sourced. It'd take a huge effort to 'move' something as entrenched as Greenplum onto PostgreSQL 9.x and the fact that so many features and enhancements are missing from 8.2, it's putting off potential users.

Re: Pivotal Greenplum Database has been open sourced

#32

Post by Herald_MJ »

polskibus wrote: post #24

What's the story behind greenplum? Is it an old startup that has been bought by pivotal some time ago? I found an article about greenplum raising 20 mln $ in 2008 [1] I was wondering whether anyone used greenplum in production and with what kind of loads. With today's in-memory fad, I am also interested in whether gp model supports loading everything into memory in a MOLAP fashion. [1] http://techcrunch.com/2008/01/2…

Greenplum was a startup that was acquired by EMC in 2010. Pivotal is partially an EMC venture, so I guess that's how it ended up in their hands.

Re: Pivotal Greenplum Database has been open sourced

#33

Post by chris_wot »

blumkvist wrote: post #26
nl wrote: post #15

Earlier quoted context omitted.

Greenplum is designed for datawarehousing. The analytics it is built for aren't the kind that ElasticSearch is typically used for. Greenplum/datawarehouse style analytical are things like: What is the average spend of female customers with 2 children who live in postcode AAAA or BBBB. Break it down by day of week and group by their marital status. Now for the top cohort of buyers on Mondays, give me a breakdown of th…

Hey, forgive my ignorance, I'm the guy who uses this stuff, not the guy who builds it. My understanding was that, typically, the DW is where the data gets ETLed from various places and lives in a normalized environment. Then from that big pool, departments get built columnar or OLAP databases for specific analyses they want to do, and I connect not to the DW but to the OLAP instance (we call it Datamart here). Was I…

It can be done with a dimensional scheme or with a normalised schema. Ralph Kimble argues for the dimensional approach whilst Bill Imon argues for the normalised approach.

For myself, I tend to go with Kimble, but there are advantages and disadvantages to both approaches. Wikipedia has a decent summary of this here:

https://en.m.wikipedia.org/wiki/Data_warehouse#Dimensional_v...

Re: Pivotal Greenplum Database has been open sourced

#34

Post by iheartmemcache »

jacques_chester wrote: post #10

Edit: There's a shiny website too -- http://greenplum.org/ We (Pivotal, for whom I don't speak in any official capacity) have also opensourced Apache HAWQ (incubating)[1], which is an SQL front-end for Hadoop that was extracted from Greenplum, as well as Apache Geode (incubating)[2] which was based on GemFire. This was part of a general announcement we made in February that our intention was to opensource our data pr…

Oh man this is huge. Are you guys opening Chorus and the Pivotal HD stuff too? I'd be shorting HP stock right now, because Vertica just lost all of its appeal. Teradata and it's Hadoop H-SQL or whatever must be shaking in their boots too. Are you guys going to attempt to upstream this or is it forked to the point of no return?

I'd love to work on a project like this but I've got no PGSQL experience (though I have worked on HFT both with KDB+ and writing a fairly decent functional knock-off operating in production and I bet the distribution and data set problems were similar). How is EMC to work for as a parent company?

Re: Pivotal Greenplum Database has been open sourced

#35

Post by alexatkeplar »

jacques_chester wrote: post #12
lobster_johnson wrote: post #11

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving t…

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Is there an existing library (like a Cascading Tap) for parallel loading Greenplum from Hadoop? I couldn't find one.

Re: Pivotal Greenplum Database has been open sourced

#36

Post by fidget »

LittlePeter wrote: post #25
piggybox wrote: post #19

would like to see some benchmark vs Redshift, though the latter is a blackbox

What do you mean blackbox? It has explain analyze, it has rich query meta info, it has a web interface to query stats. You even know on what hardware it runs. Genuinely not sure what's blackbox about Redshift.

We had a cluster last week spend 7 hours in a DB health 'unknown' state. Response from AWS? Spin up a new cluster from snapshot.

Re: Pivotal Greenplum Database has been open sourced

#37

Post by moretti »

__david__ wrote: post #27

What's up with just a single commit since 2006? https://github.com/greenplum-db/gpdb/commits/master Why include all the really old commits, while squashing the most recent 10 years of commits into a single commit (6b0e52bead)?

Reminds me of this tweet:

https://twitter.com/capotej/status/647213485294616576

Re: Pivotal Greenplum Database has been open sourced

#38

Post by jacques_chester »

x0x0 wrote: post #23
lobster_johnson wrote: post #13

Earlier quoted context omitted.

Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the d…

I used greenplum until about three years ago when that employer replaced it. There is no way to describe it other than brittle and unsuited for production work. My peers discovered multiple ways to kill the db, all with data loss. Before you use it you should find a current user. It is one of a handful of techs on my personal "never again" list. Licensing was also ludicrously expensive, including an attempt to multip…

> I'd be very leery of lock-in given the owners.

The owner is now Pivotal (it used to belong to EMC). Almost all our product code is opensource. The culture comes from Pivotal Labs DNA, which is a resolutely opensource shop.

> I also suspect this is the result of a failed attempt to sell it and is an open-source hail mary.

It's not.

Re: Pivotal Greenplum Database has been open sourced

#39

Post by jacques_chester »

iheartmemcache wrote: post #34
jacques_chester wrote: post #10

Edit: There's a shiny website too -- http://greenplum.org/ We (Pivotal, for whom I don't speak in any official capacity) have also opensourced Apache HAWQ (incubating)[1], which is an SQL front-end for Hadoop that was extracted from Greenplum, as well as Apache Geode (incubating)[2] which was based on GemFire. This was part of a general announcement we made in February that our intention was to opensource our data pr…

Oh man this is huge. Are you guys opening Chorus and the Pivotal HD stuff too? I'd be shorting HP stock right now, because Vertica just lost all of its appeal. Teradata and it's Hadoop H-SQL or whatever must be shaking in their boots too. Are you guys going to attempt to upstream this or is it forked to the point of no return? I'd love to work on a project like this but I've got no PGSQL experience (though I have wor…

> Are you guys opening Chorus and the Pivotal HD stuff too?

I'm not sure where Chorus and PHD fit into the picture -- the latter is I believe a distribution of Hadoop that we curate.

> Are you guys going to attempt to upstream this or is it forked to the point of no return?

Too early to say.

> How is EMC to work for as a parent company?

EMC is entirely hands off, in my experience. Pivotal runs itself. Best job I've ever had by a wide margin.

Re: Pivotal Greenplum Database has been open sourced

#40

Post by jacques_chester »

alexatkeplar wrote: post #35
jacques_chester wrote: post #12

Earlier quoted context omitted.

> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of herois…

Is there an existing library (like a Cascading Tap) for parallel loading Greenplum from Hadoop? I couldn't find one.

You should be able to build a pipeline with Spring XD, it has an HDFS source and gpfdist sink out of the box.

http://projects.spring.io/spring-xd/

Post reply on HN