Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redshift Performance and Cost

nerds.airbnb.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Redshift Performance and Cost

#41

Post by jacques_chester »

eru wrote: post #40
jacques_chester wrote: post #39

Earlier quoted context omitted.

I'm not sure I follow you.

To be more stereotypical: I am intrigued and would like to sign up to your newsletter.

Well ... really, just read a good pair of textbooks on each side of the spectrum. Date's Databases and Kimball's The Data Warehouse Toolkit are good.

Edit: actually, maybe not Date. It's up to you. It's good, but it's controversial because he's not a fan of SQL and so he uses his own language.

The one I used in uni was Ramakrishnan & Gehrke's Database Management. It was OK but there's a certain amount of at-the-time trendy bullshit that to me detracts from a focus on relational databases for their own sake.

Edit 2: and Joe Celko's SQL for Smarties contains good oil on the relational paradigm.

Re: Redshift Performance and Cost

#42

Post by wisty »

eru wrote: post #38
jacques_chester wrote: post #35

Earlier quoted context omitted.

I realised a few years ago that pretty much every database course taught only teaches OLTP. OLAP never really gets a lookin. At my university, standard normalisation was taught in the "databases" course. OLAP was mentioned as part of the "advanced databases" course. The database course at that time blew about half its time on building PHP applications to talk to the database. I hate to second guess my professors, but…

Do you know of any resource that talks about the same topic as your comment, but in more detail?

Philip Greenspan's SQL tutorial is a nice starting point: http://philip.greenspun.com/sql/

It's a bit old, but still pretty good.

Re: Redshift Performance and Cost

#43

Post by tszming »

alexatkeplar wrote: post #24
nieksand wrote: post #12

Earlier quoted context omitted.

I think part of the issue why so many people have gone with Hive is that good, production-ready column stores are expensive. Redshift is posed to change that. If you're shopping in this space, Infobright is also worth checking out. And even for moderate data sizes (10+ GB per table), row store DBs tend to become painful. This is especially true when you need to support ad-hoc reporting queries, since the usual techni…

We use Infobright at SnowPlow ( https://github.com/snowplow/snowplow ), and are currently working on our Redshift integration. One thing to be aware about with both is the lack of any support for wide tables - Infobright inherits MySQL's limit of 65,535 bytes per row (and UTF8 means 3 bytes per char); with Redshift you can stored wider rows but you can't query them ( http://docs.aws.amazon.com/redshift/latest/dg/r_CR…

Hi, I have started your project previously and while I haven't had a chance to test it out, I must say the idea of using cloud-front as a collector is a superb brilliant idea to scale an analytics platform and would be both very scalable, reliable and economical.

Btw, do you have more experience to share? e.g. with infobright, how many events can be processed per second? what would be the "ETL latency"? can infobright handle 10TB of data easily, any caveat besides the row limit? Thanks.

Re: Redshift Performance and Cost

#44

Post by bloomfilter »

TY wrote: post #5

Redshift is based on ParAccel, not on Postgres. ParAccel uses APIs similar to Postgres due to historical reasons, but not the technology. For a basic overview: http://en.wikipedia.org/wiki/Paraccel As for the rest of the article, it feels like a basic Data Warehousing 101 re-discovered. It should have been titled "Analytics: Back To The Future" :-)

Thanks for pointing it out, we have correct it in our post

Re: Redshift Performance and Cost

#45

Post by bloomfilter »

polskibus wrote: post #11

It's good to see a Redshift evaluation. I'm wondering how does Redshift compare to hadoop airbnb setup when taking data loading and transformation into consideration as well as running aggregate queries? I mean if you want to run analysis fairly often, do you need to reload everything in Redshift? From maintenance point of view, is Hadoop setup more flexible and cheaper than Redshift?

The hadoop setup we have is actually EMR, and we use s3 for data storage, so for us Hive/Hadoop doesn't save much in terms of data loading. And once you have a process setup to load and update the data in Redshift, you don't have to reload everything

Re: Redshift Performance and Cost

#47

Post by alexatkeplar »

tszming wrote: post #43
alexatkeplar wrote: post #24

Earlier quoted context omitted.

We use Infobright at SnowPlow ( https://github.com/snowplow/snowplow ), and are currently working on our Redshift integration. One thing to be aware about with both is the lack of any support for wide tables - Infobright inherits MySQL's limit of 65,535 bytes per row (and UTF8 means 3 bytes per char); with Redshift you can stored wider rows but you can't query them ( http://docs.aws.amazon.com/redshift/latest/dg/r_CR…

Hi, I have started your project previously and while I haven't had a chance to test it out, I must say the idea of using cloud-front as a collector is a superb brilliant idea to scale an analytics platform and would be both very scalable, reliable and economical. Btw, do you have more experience to share? e.g. with infobright, how many events can be processed per second? what would be the "ETL latency"? can infobrigh…

Hi tszming - happy to share more over email. alex@snowplowanalytics.com
Post reply on HN