Airbnb open-sources Caravel: data exploration and visualization platform • Hacker News Board

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#41

Post by shostack » Thu, Mar 31, 2016, 8:34 PM UTC

teej wrote: post #19

caravel wrote: post #16

Earlier quoted context omitted.

We'll be providing short user training videos very soon.

As someone who is the core audience for this tool can I say that I strongly prefer clear documentation over videos? Videos are way too hard to maintain and end up being stale the minute after you post them in fast-moving projects. I can't text-search a video and I can't be linked directly to an answer in a StackOverflow response. Written documentation is vastly superior to videos in my opinion.

To add--you can also skim text MUCH more quickly for the piece you want compared to video.

I absolutely loathe video for any analytics-related documentation. It rarely adds any real value over text outside of live webinars where I can ask questions.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#42

Post by ngould » Thu, Mar 31, 2016, 8:34 PM UTC

Rapzid wrote: post #23

jonbishop wrote: post #17

Earlier quoted context omitted.

Hey! I run marketing for Periscope Data, a data explorer/dashboard product. We have a lot of customers using postgres and get compared to Looker a lot. We focus on optimizing for the analyst whereas tools like Looker focus on business users. We have a lot of features for business users, but chart creation is all SQL based. Our site is here: https://www.periscopedata.com/ and if you have any questions, shoot me an ema…

Do you have a self-hosted option? What's your pricing?

Word on the street is Periscope costs $1000/mo for unlimited users, up to 1B rows.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#43

Post by maerF0x0 » Thu, Mar 31, 2016, 9:31 PM UTC

twakefield wrote: post #21

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

Few more for your list: Leftronic (service) or Grafana (package)

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#44

Post by vhhuhhfryuhgfh » Thu, Mar 31, 2016, 9:31 PM UTC

Any pictures anywhere?

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#45

Post by ultimoo » Thu, Mar 31, 2016, 9:31 PM UTC

don_draper wrote: post #15

The code is so clean and simple. This is great PR for the company. I want to work there.

I think so too!

I have done a fair bit of Ruby a few years ago but I'm new to python CRUD apps and trying to improve my knowledge here. Is defining all models in the same file[1] conventional in python apps? Rails used to have separate files for each model. And most Ruby apps that I have seen advocate the one-class-one-file convention.

[1] https://github.com/airbnb/caravel/blob/master/caravel/models...

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#46

Post by adrr » Thu, Mar 31, 2016, 9:38 PM UTC

phunge wrote: post #30

twakefield wrote: post #21

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.

Looker performance comes from the underlying storage of data. You can store massive amounts of data on say a Red Shift cluster and still be performant. Most visualization tools that i know are directly tied to storage tier in terms of performance.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#47

Post by caravel » Thu, Mar 31, 2016, 9:44 PM UTC

dwmintz wrote: post #38

phunge wrote: post #30

Earlier quoted context omitted.

They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.

It's worth distinguishing between the tools that leave the data in your data warehouse (Caravel, Periscope, Mode, Looker (where I work)), and those that have their own data stores (Good Data, Qlikview, etc.) Tableau can connect directly to your datastore, but it's happier if it can operate on data that's stored locally in-memory. Anyway, the ones where bring your own database can scale as far as the database can brin…

Note that Tableau doesn't play well with Presto which Airbnb uses extensively. No possibility of using the "live mode"

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#48

Post by caravel » Thu, Mar 31, 2016, 9:45 PM UTC

adrr wrote: post #46

phunge wrote: post #30

Earlier quoted context omitted.

They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.

Looker performance comes from the underlying storage of data. You can store massive amounts of data on say a Red Shift cluster and still be performant. Most visualization tools that i know are directly tied to storage tier in terms of performance.

I don't think Looker supports Druid.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#49

Post by polskibus » Thu, Mar 31, 2016, 9:52 PM UTC

I'm wondering - what's the effort required to build such a BI tool (tables, charts, maps) these days assuming reusing open source components and focusing on SQL-speaking datastores? Could a small team of experienced devs accomplish such a feat in a year?

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#50

Post by caravel » Thu, Mar 31, 2016, 9:57 PM UTC

twakefield wrote: post #21

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

Well none of the solutions mentionned are open source.

Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can.

More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire quarters or years for Tableau to parallelize their "live mode", or to get connectivity to Presto to work?

What if you want to integrate a new type of visualization that isn't supported? What if you want to integrate with your anomaly detection framework or your A/B testing framework or other internal or external facing applications?

Since this is a common need for most companies, it makes sense to have an open source solution that we can all use and collaborate on.