61–70 of 95 posts
Grouping imports into: standard lib, third party, local is a strong pattern that I don't see done consistently in many repos. Likewise with your use of wrapping long imports with ()s and a single tab.
Any chance of sharing your Python style guide? My startup is Python based (Django and Flask) and would really appreciate it!
Awesome - great work! A tutorial on how to link it to a mysql database would be greatly appreciated :)
So, you just need to set the config param SQLALCHEMY_DATABASE_URI like this:
https://github.com/airbnb/caravel/blob/1b4e750b2aa111445703d...
The configuration guide explains it further:
https://github.com/airbnb/caravel/blob/master/docs/installat...
what are the supported data sources? I saw SQL tables and I imagine flat files, can you write calls to web service endpoints?
From the readme file: Database Support Caravel was originally designed on top of Druid.io, but quickly broadened its scope to support other databases through the use of SqlAlchemy, a Python ORM that is compatible with most common databases[1]. [1] http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html
If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that…
Disclaimer: I work on Metabase.
Earlier quoted context omitted.
Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire…
"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…
Building a community can also help distributing the costs. But more importantly it's a labour of collaboration instead of a client/vendor struggle.
Earlier quoted context omitted.
"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…
There's also an in house cost in maintaining vendor software and integrating it. That's on top of the 20% usual maintenance fee. Building a community can also help distributing the costs. But more importantly it's a labour of collaboration instead of a client/vendor struggle.
But if you find this to be a frequent occurrence, you're almost certainly evaluating costs incorrectly.
Earlier quoted context omitted.
Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire…
"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…
We also use some of the previously mentioned visualization technologies and get generally poor results by all measures.
If AirBNB has a few engineers around who are part of solving a unique challenge AirBNB has, then they are probably getting a better deal than we are. I'd wager their productivity per resource is significantly better, not to mention licensing, infrastructure and control over product features. That said, not everyone is AirBNB and has the ability to attract and retain that type of talent.
Earlier quoted context omitted.
"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying i…
I work at a Fortune 100 and we use fairly well known warehousing database. I can tell you we spend a serious amount of money for our licenses, infrastructure, employee talent, staff-aug contractors and management overhead running this commercial solution. Much of the human resource I would classify as over compensated and non-productive. We also use some of the previously mentioned visualization technologies and get…
I'm not saying that you wouldn't choose a good open source alternative if it exists. I'm saying that you shouldn't run off and implement something that exists commercially if your only reason is that the commercial thing costs money.
Earlier quoted context omitted.
Can't speak for Airbnb, but I'm not sure that any of the front-end clients that you mentioned (disclosure: I work at Looker), can talk to Druid. So if Airbnb already had a Druid warehouse in place, they may have decided it was easier to roll their own front-end than migrate to a different backend.
Druid is definitely part of the equation. Larger, data-driven companies with significant engineering teams prefer not relying on 3rd party, closed-sourced vendors. That can represent a significant risk and a blockage for deeper integration with other internal applications when needed. Not that building always wins over buying, but the balance shifts relatively to the size of the company. Also, when using open source…
I work on Google Cloud, where we have both closed-source - BigQuery, open source - Dataproc, and closed source that makes open source rock - Dataflow/Beam. There are merits to each.
BigQuery is serverless and multi-tenant and can't really exist outside of Google Cloud due to its intrinsic dependency on low-level services that don't exist elsewhere (and for the same reasons we couldn't directly externalize Borg, choosing to create an OSS clone in Kubernetes).
It is not unusual for me to hear from folks that they spent 6+ months building a performant and sizable, say Presto cluster. Then there's continuous management, tinkering, configuration, and optimization projects. I hear this from companies that one would consider sophisticated technologically.
By contrast, 40% of all of BigQuery's Petabyte customers scale to these levels without ever talking to us. We just find them on consumption reports. On multiple occasions we've had "surprise" load tests of millions of rows per second streamed into BigQuery, and it just works. BigQuery is also HA out of the box at no additional cost, which is a great luxury.
So sometimes if you need to scale analytics to Petabytes, the option is to just consume a managed and cost-effective service, or tinker with OSS, where there are significant operational tradeoffs. On the other hand, as you said, you build pride and culture. It's also far from an automatic shoo-in that OSS gives you better TCO (against the old closed-source guard, yes, but not so much BQ). Thus, the relationship between company size and value of technology can invert at higher levels.
With all that, I'd love to see a Caravel-BQ plugin :)
(PS. Kudos to Druid for introducing a Streaming ingest. BigQuery also sees value in Streaming ingest, GA-ing our own Streaming API in March of 2015, and Kudos to Airflow).