Moving product recommendations from Hadoop to Redshift saves us time and money

Post by mattj » Thu, Jun 19, 2014, 2:00 PM UTC

endersshadow wrote: post #19

mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!

Post by mattj » Thu, Jun 19, 2014, 2:04 PM UTC

mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Post by noir_lord » Thu, Jun 19, 2014, 2:24 PM UTC

mattj wrote: post #21

endersshadow wrote: post #19

Earlier quoted context omitted.

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!

Yup! My experience with redshift has actually made me curious to try out Postgres (I've always used MySQL before this). The stricter SQL dialect was a little odd at first, but I think I've become more comfortable with it over a few months.

Post by hcho » Thu, Jun 19, 2014, 2:34 PM UTC

personZ wrote: post #18

onion2k wrote: post #14

Earlier quoted context omitted.

It's often cheaper to use the "wrong" architecture than optimising the right architecture. I know I could have a CouchDB datastore searching a few GB with an afternoon of work. I imagine I could get MySQL fast enough with a few days of optimisation. In terms of time, which is by far the biggest cost in most development, CouchDB is the better option.

In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…

Post by CurtHagenlocher » Thu, Jun 19, 2014, 2:42 PM UTC

endersshadow wrote: post #19

mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!