Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scuba: Diving into Data at Facebook [pdf]

research.fb.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Scuba: Diving into Data at Facebook [pdf]

#21

Post by spicyj »

bajsejohannes wrote: post #19

Is this replacing the Gorilla database? Or is it using it under the hood? Or do they co-exist? If so, how are they used differently? For those who don't know, the Gorilla database is also from Facebook and they published a paper about it roughly a year ago: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

They're complementary. The ODS system mentioned there is more for monitoring numerical metrics (like Graphite) and doesn't support logging arbitrary data.

ODS is good for top-line metrics and can handle more volume but doesn't compare to Scuba if you want to dig in and look at individual rows in your data (or even just analyze your data and group based on certain columns).

Re: Scuba: Diving into Data at Facebook [pdf]

#23

Post by _fsjdf_ »

georgewfraser wrote: post #5

This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by using a standard data warehouse like Redshift or BigQuery, a queue like Kinesis, and a BI tool like Looker or Tableau. Your data won't be quite as real-time and your queries won't be quite as fast, but it will take much less engineering effort and you'll be abl…

Wouldn't InfluxDB be a better comparison?

Re: Scuba: Diving into Data at Facebook [pdf]

#24

Post by siliconc0w »

georgewfraser wrote: post #5

This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by using a standard data warehouse like Redshift or BigQuery, a queue like Kinesis, and a BI tool like Looker or Tableau. Your data won't be quite as real-time and your queries won't be quite as fast, but it will take much less engineering effort and you'll be abl…

Here is a good list of redshift 'gotchas' - https://github.com/open-guides/og-aws#redshift-gotchas-and-l...

We ran into several of those. Notably, it's difficult to achieve the 'real-time' promise of redshift because of the huge performance hit while loading data into the DB so you have to do it off-hours. You can update a replica and then 'hot-swap' it in but this gets expensive. For operational analytics it's better to go with one of the purpose-built timeseries databases and dual write to that and your data warehouse.

Re: Scuba: Diving into Data at Facebook [pdf]

#26

Post by lstyls »

buremba wrote: post #12

How does Scuba differ from Presto which is also developed by Facebook? It seems that it stores data in-memory and have data expiration feature but also has many common features such as SQL and distributed processing.

Scuba made decisive tradeoffs in the functionality that it provides. Notable ones include that it doesn't support joins within a table, and doesn't provide any cross-table operations. Mostly it is used for basic filtering on constant values, and gathering summary statistics on those values. This is less of a limitation than it sounds like because when you know this ahead of time you just log your writes in a denormalized way and you don't need to join anything later.

As @ot said, Presto is just a query engine and it doesn't provide a backend. It provides an API that allows it to be plug in to different data warehousing systems. I would assume functionality depends to some extent on what your data is stored in, but in general Presto supports the full suite of standard relational db style queries.

Source: I work at FB as well. In fact I was using Scuba just now to do a quick analysis of our storage requirements for Scuba itself :)

Re: Scuba: Diving into Data at Facebook [pdf]

#28

Post by lstyls »

_fsjdf_ wrote: post #22

How does this compare to Splunk on machine data or Tableau on relational data?

I don't know those platforms, but Scuba doesn't support relational operations. It's not as much of a limitation as it sounds like because you can log your data in a denormalized way.

Re: Scuba: Diving into Data at Facebook [pdf]

#29

Post by nbm »

_fsjdf_ wrote: post #22

How does this compare to Splunk on machine data or Tableau on relational data?

Comparing to Tableau, Scuba is schema-less and doesn't require any setup beyond creating the table (which doesn't have any approval process in the way and gives you a reasonable amount of scratch space to test before you get serious) and then having data arrive. Once Scuba is aware of your column by you submitting data for it, it allows you to query/group based on it very quickly. Scuba is entirely real-time. There are some other projects to do pre-computed aggregates if that's important to you.

I haven't used Splunk in ~7 years, so I can't remember enough about it to compare well. Splunk has some structured-on-top-of-unstructured stuff in it, whereas Scuba is always structured. If you want to turn something unstructured into something structured, you generally run a separate pipeline to do that (using one of the tailing frameworks for your preferred language). In terms of the alarm system in Splunk, we have other systems for handling that using the data that flows into Scuba.

Re: Scuba: Diving into Data at Facebook [pdf]

#30

Post by liorabraham »

There's also a blog post about it here https://www.facebook.com/notes/facebook-engineering/under-th...

My YC company https://www.interana.com took a lot of lessons from this and is doing something that I think is even better for cos like Reddit, Sonos, Comcast, Bing. You can sign up if you'd like a demo :)

Post reply on HN