Former Scuba dev, Okay Zed, wrote http://snorkel.logv.org/. It's open source.
Scuba: Diving into Data at Facebook [pdf]
11–20 of 44 posts
Re: Scuba: Diving into Data at Facebook [pdf]
#12How does Scuba differ from Presto which is also developed by Facebook? It seems that it stores data in-memory and have data expiration feature but also has many common features such as SQL and distributed processing.
Re: Scuba: Diving into Data at Facebook [pdf]
#13How does Scuba differ from Presto which is also developed by Facebook? It seems that it stores data in-memory and have data expiration feature but also has many common features such as SQL and distributed processing.
Scuba is a complete system of log collection, storage and retrieval, and UI/visualization.
Presto would only cover the storage/retrival part. Scuba has its own backend for that which is very optimized for the kind of queries the UI needs to support, while Presto is a generic SQL store for analytics.
Re: Scuba: Diving into Data at Facebook [pdf]
#14This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by using a standard data warehouse like Redshift or BigQuery, a queue like Kinesis, and a BI tool like Looker or Tableau. Your data won't be quite as real-time and your queries won't be quite as fast, but it will take much less engineering effort and you'll be abl…
Shameless plug: I'm the maintainer of Rakam project.
You don't have to deal with all of these complexities. There are open-source projects that can setup that infrastructure for you. For example, we provide Cloudformation scripts that setup data analytics cluster for you, Kinesis, S3, PrestoDB and RESTFul API for collecting and querying data-sets so that you can use it similar to how you use a SaaS product. We also have an integrated visualization project that can connect your Rakam API and allow you to create dynamic reports and custom dashboards. You can run queries via user interface similar to Scube UI and also complex behavioral queries such as funnel and cohort queries as well as SQL. https://github.com/rakam-io/rakam
Re: Scuba: Diving into Data at Facebook [pdf]
#15Former Scuba dev, Okay Zed, wrote http://snorkel.logv.org/ . It's open source.
Is there a comparison to scuba?
Re: Scuba: Diving into Data at Facebook [pdf]
#16How does Scuba differ from Presto which is also developed by Facebook? It seems that it stores data in-memory and have data expiration feature but also has many common features such as SQL and distributed processing.
Scuba is a complete system of log collection, storage and retrieval, and UI/visualization. Presto would only cover the storage/retrival part. Scuba has its own backend for that which is very optimized for the kind of queries the UI needs to support, while Presto is a generic SQL store for analytics.
How does Scuba optimize the stored data-sets for aggregation queries compared to Presto (Raptor connector)? They both use common columnar data storage techniques such as compression, delta encoding and dictionary encoding. The main difference seems to be the real-time nature of Scuba and the UI.
Re: Scuba: Diving into Data at Facebook [pdf]
#17I'm so happy to see this posted. I've been at Facebook for two years, and Scuba is hands-down one of my most favorite internal tools we have (and we have a lot of good ones). The article focuses a lot on the implementation, but thankfully you don't need to worry about that when using it. The flexible/quick/easy UI is what seals the deal for me, combined with the fast query times and realtime data -- you can use it to…
I really miss the awesome tooling like Scuba since I left.
I'll second the post above -- if you miss Scuba, honeycomb.io is for you. https://honeycomb.io/blog/2016/11/honeycomb-faq-in-140-chars...
Re: Scuba: Diving into Data at Facebook [pdf]
#18Earlier quoted context omitted.
Scuba is a complete system of log collection, storage and retrieval, and UI/visualization. Presto would only cover the storage/retrival part. Scuba has its own backend for that which is very optimized for the kind of queries the UI needs to support, while Presto is a generic SQL store for analytics.
How does Scuba optimize the stored data-sets for aggregation queries compared to Presto (Raptor connector)? They both use common columnar data storage techniques such as compression, delta encoding and dictionary encoding. The main difference seems to be the real-time nature of Scuba and the UI.
Oh yeah good point, I had forgotten that Presto does not support realtime. About optimizations, I don't know the details, but for one, Scuba is C++ and Presto is Java.
Re: Scuba: Diving into Data at Facebook [pdf]
#19Is this replacing the Gorilla database? Or is it using it under the hood? Or do they co-exist? If so, how are they used differently?
For those who don't know, the Gorilla database is also from Facebook and they published a paper about it roughly a year ago: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
Re: Scuba: Diving into Data at Facebook [pdf]
#20Yay!! So happy to see Scuba get more visibility. I can't even count the number of times I said, and heard other engineers say, that the thing they would miss most about Facebook was Scuba.
That's the whole reason why we built honeycomb.io. If you're a FB expat, check us out.