Live data from Hacker News

Airbnb open-sources Caravel: data exploration and visualization platform

github.com

71–80 of 95 posts

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#71
post #54
post #50

Earlier quoted context omitted.

Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire…

"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…

> Developers have a real, well-documented resistance to paying for things

This is why you never try to target developers and if you do, you have to make sure the initial cost to use it is negligible. This is what Atlassian does very well, with their pricing structure.

In the past, I developed in house tools for massive Enterprise companies and the common theme has always been, we can do it better and cheaper. Sometimes that's true, sometimes it's not. I noticed this trend slowly tapering off before I left to start my startup and this was due to the availability of better open source solutions.

Technical people always overestimate what they are capable of, present company included, and if you are set on selling to developers, you have to make sure the barrier to use is negligible. By at least trying, they will either come to the conclusion that what you have created is a piece of shit or they realize it's not worth their time and effort to duplicate/maintain.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#72
post #29

Earlier quoted context omitted.

From the readme file: Database Support Caravel was originally designed on top of Druid.io, but quickly broadened its scope to support other databases through the use of SqlAlchemy, a Python ORM that is compatible with most common databases[1]. [1] http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html

Specifically, SQLAlchemy includes dialects out of the box for: Firebird, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite, Sybase. http://docs.sqlalchemy.org/en/rel_1_0/dialects/index.html

And has adapters for Hive, Presto, Redshift and Google BigQuery.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#75
post #51

I'm wondering - what's the effort required to build such a BI tool (tables, charts, maps) these days assuming reusing open source components and focusing on SQL-speaking datastores? Could a small team of experienced devs accomplish such a feat in a year?

Yes, look at the commit log. https://github.com/airbnb/caravel/graphs/contributors

Is it covering the very beginning when the caravel wasn't open source?

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#77
post #69
post #67

Earlier quoted context omitted.

I work at a Fortune 100 and we use fairly well known warehousing database. I can tell you we spend a serious amount of money for our licenses, infrastructure, employee talent, staff-aug contractors and management overhead running this commercial solution. Much of the human resource I would classify as over compensated and non-productive. We also use some of the previously mentioned visualization technologies and get…

But what's the alternative? Write your own warehousing database? Likely not. Even AirBnB hasn't done that. I'm not saying that you wouldn't choose a good open source alternative if it exists. I'm saying that you shouldn't run off and implement something that exists commercially if your only reason is that the commercial thing costs money.

Yeah build our own isn't a good option for us for a few reasons.

1. Our problem set is not novel, we're more or less like many others out there and collectively we are a market that a vendor can build a product for

2. Since we have significantly more revenue than a startup like AirBNB we can rationalize and amortize an ongoing expense

3. We actively try not to do things that aren't our core competency

4. Even if we wanted to we have plenty of human resources but not the talent to build our own analytic DB or data visualization tools

I think AirBNB is likely a much different company than most. Their approach to talent attraction/retention, novelty of problem set, ability and tolerance to make a multimillion dollar capital investment in analytic database and visualization tools, etc, etc. In their case the upstart cost of building a custom thing and the value prop likely makes sense.

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#78

I really like the Python style you guys have adopted. Grouping imports into: standard lib, third party, local is a strong pattern that I don't see done consistently in many repos. Likewise with your use of wrapping long imports with ()s and a single tab. Any chance of sharing your Python style guide? My startup is Python based (Django and Flask) and would really appreciate it!

The import ordering is pretty common: https://google.github.io/styleguide/pyguide.html?showone=Imp...

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#79

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…

investor's money, baby!

Re: Airbnb open-sources Caravel: data exploration and visualization platform

#80
Hey, this is very interesting, I will take a look. I am trying to shift a whole $700m division and then hopefully $3b segment on a new workflow paradigm for data (automatically refreshed dashboards instead of sending around excel files, focus on building models and not data pasting in spreadsheets) and unfortunately for now Tableau is my only option. I feel very uncomfortable going with a closed solution since I know that we will have lots of edge cases and that being able to do your own coding is in the end the best way to deal with those. License cost is also incredibly high, we are talking $200 per user per year at minimum, that means 200 to 400 thousand per year for a big organization between 1000 and 200 users.
Post reply on HN