Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airbnb open-sources Caravel: data exploration and visualization platform

github.com

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#81

Post by gedrap »

ultimoo wrote: post #45
don_draper wrote: post #15

The code is so clean and simple. This is great PR for the company. I want to work there.

I think so too! I have done a fair bit of Ruby a few years ago but I'm new to python CRUD apps and trying to improve my knowledge here. Is defining all models in the same file[1] conventional in python apps? Rails used to have separate files for each model. And most Ruby apps that I have seen advocate the one-class-one-file convention. [1] https://github.com/airbnb/caravel/blob/master/caravel/models...

It depends on the project.

If you have a small number of models (e.g. When your application is growing, you have split the models into multiple files, grouped by features, etc (e.g. users.py, content.py, etc).

I prefer this as models usually a very small, and switching from one file to another can become quickly annoying when working on related models. However, it may be different for large classes.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#82

Post by gedrap »

tlrobinson wrote: post #64
twakefield wrote: post #21

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

To throw out an open source one: Metabase http://www.metabase.com https://github.com/metabase/metabase Disclaimer: I work on Metabase.

Not really related to your product, but I find the trend of using emojis in commit messages really distracting. It's probably okay-ish very very rarely for something huge, but commit log reading like slack chat... Em not sure about it.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#83

Post by nnq »

timr wrote: post #54
caravel wrote: post #50

Earlier quoted context omitted.

Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire…

"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…

> than paying someone to maintaining it yourself

You do realize that this is probably THE reason to open-source a piece of custom software after you've built it, right?

...I imagine that they expect the community will do X% of the maintenance work now, so they can save Y%, so win-win for both the company and the community. Also, you increase the probability that any new hires will be at least mildly familiar with your in-house stack if you open-source some of it (think TensorFlow).

Free as in beer is the best answer when you have reasonable expectations that the community will really embrace the product and you'll get back "free as in beer" upgrades and bugfixes to the software you won't otherwise have the budget to properly maintain (and properly document, btw! think of all the free tutorials that will be written for this after it gets popular!) :)

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#85

Post by ernestbro »

twakefield wrote: post #21

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

There is a study done by a market analyst (Wayne Eckerson) about Building vs Buying BI, he has some good insights based on surveys as to why some companies chose to build and some choose to buy

71% of those who chose to build the BI tools said they built because "We can customize the functionality better"

51% of those who buy say "Buying enables us to provide best-in-class BI functionality"

The study: http://www.jaspersoft.com/sites/default/files/confirmation_f...

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#86

Post by CurtMonash »

vgt wrote: post #70
caravel wrote: post #53

Earlier quoted context omitted.

Druid is definitely part of the equation. Larger, data-driven companies with significant engineering teams prefer not relying on 3rd party, closed-sourced vendors. That can represent a significant risk and a blockage for deeper integration with other internal applications when needed. Not that building always wins over buying, but the balance shifts relatively to the size of the company. Also, when using open source…

Your points resonate with me very strongly, but I do have a counter argument. I work on Google Cloud, where we have both closed-source - BigQuery, open source - Dataproc, and closed source that makes open source rock - Dataflow/Beam. There are merits to each. BigQuery is serverless and multi-tenant and can't really exist outside of Google Cloud due to its intrinsic dependency on low-level services that don't exist el…

Roughly how many petabyte customers are you talking about?

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#87

Post by rufugee »

dacort wrote: post #40
javiercr wrote: post #20

Earlier quoted context omitted.

Take a look to http://www.metabase.com/ (open source, made by Expa, Uber's co-founder incubator).

Second this recommendation. Very easy to get up and running - may not be able to handle some complex use cases, but for the basics it's fantastic.

It must have come very far in the past few months then...based on my experimentation, it was far from being ready for production use if you needed anything beyond the most basic of SQL queries. That said, I'm rooting for it (and now, caravel). We use Pentaho, and it's....meh. A lot of moving parts, a big learning curve, and a user experience that's just ok. I'm hoping Metabase (and now Caravel) will both succeed...the open source BI space could really use an influx of simpler, beautiful tools.

A side note: I love that Caravel is written in Python. Metabase switched from Python to Clojure, and I personally believe that to be a barrier to entry for contributing. I've started down the Clojure path a number of times only to stop because I see it as something which would be difficult to impose on my team...Lisp is just so different from what most enterprisey development teams are used to. Python, on the other hand, is easier to justify. I've found things I wanted to help fix in Metabase, but having to learn idiomatic Clojure just to submit a patch is a turn-off.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#88

Post by dwmintz »

caravel wrote: post #48
adrr wrote: post #46

Earlier quoted context omitted.

Looker performance comes from the underlying storage of data. You can store massive amounts of data on say a Red Shift cluster and still be performant. Most visualization tools that i know are directly tied to storage tier in terms of performance.

I don't think Looker supports Druid.

That's correct. We (I work at Looker) are written to leverage the analytical capabilities of SQL. So we support all kinds of SQL implementations, from totally standard (PG, MySQL, MS) to MPPs (Redshift, BigQuery) to SQL-on-Hadoop (Hive, Impala, Presto, Spark).

But the JDBC -> Druid connectors that exist look pretty janky. So if someone builds a stable connector, I suspect we'd support it. But for the moment, no Druid.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#89

Post by dwmintz »

caravel wrote: post #47
dwmintz wrote: post #38

Earlier quoted context omitted.

It's worth distinguishing between the tools that leave the data in your data warehouse (Caravel, Periscope, Mode, Looker (where I work)), and those that have their own data stores (Good Data, Qlikview, etc.) Tableau can connect directly to your datastore, but it's happier if it can operate on data that's stored locally in-memory. Anyway, the ones where bring your own database can scale as far as the database can brin…

Note that Tableau doesn't play well with Presto which Airbnb uses extensively. No possibility of using the "live mode"

Not arguing that Airbnb should use Looker, but fwiw, Looker (where I work) does in fact connect to Presto fine.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#90

Post by dwmintz »

kfk wrote: post #80

Hey, this is very interesting, I will take a look. I am trying to shift a whole $700m division and then hopefully $3b segment on a new workflow paradigm for data (automatically refreshed dashboards instead of sending around excel files, focus on building models and not data pasting in spreadsheets) and unfortunately for now Tableau is my only option. I feel very uncomfortable going with a closed solution since I know…

I'm a huge proponent of the idea of centralizing the data model. That's the core idea behind Looker (where I just came to work after 3 years as a customer), and I agree it's a hugely powerful change from the world of everybody-in-their-own spreadsheet.

On your other point, though, to echo the build vs. buy discussion from above, I think it's a bit misleading to say "oh, we'll just use an open-source solution and that'll be cheaper." Because if open source means a couple of internal developers and an analyst, that's easily $300k+/year in salaries that you might not spend if you were using a vendor.

Anyway, given your particular statement of the problem you're facing, I'd humbly suggest you take a look at Looker. The data modeling layer that's core to Looker is meant to solve EXACTLY that problem, by leaving your data where it lives and then embedding your business logic in the layer that sits between end users and the data.

Post reply on HN