Scuba: Diving into Data at Facebook [pdf]

Post by georgewfraser » Tue, Jan 24, 2017, 5:39 AM UTC

siliconc0w wrote: post #24

georgewfraser wrote: post #5

This is a special-purpose time-series data warehouse and a UI for querying it. If you are not Facebook, it is almost always better to do projects like this by using a standard data warehouse like Redshift or BigQuery, a queue like Kinesis, and a BI tool like Looker or Tableau. Your data won't be quite as real-time and your queries won't be quite as fast, but it will take much less engineering effort and you'll be abl…

Here is a good list of redshift 'gotchas' - https://github.com/open-guides/og-aws#redshift-gotchas-and-l... We ran into several of those. Notably, it's difficult to achieve the 'real-time' promise of redshift because of the huge performance hit while loading data into the DB so you have to do it off-hours. You can update a replica and then 'hot-swap' it in but this gets expensive. For operational analytics it's bette…

Post by debt » Tue, Jan 24, 2017, 5:47 AM UTC

nbm wrote: post #40

debt wrote: post #39

So it samples data less than or equal to a second old and at a rate determined by the person making the query? I wonder how often the data is inaccurate given the potentially low sample size?

Not sure where you got the "less than or equal to a second old"? Maybe I'm misunderstanding what you mean? There is no single system-wide imposed sampling rate, so it's up to you to set the sampling rate based on what sort of queries you want to be able to do with good enough accuracy. We have 1:1 rate data for some things (say errors served on a particular service), while a ten or a hundred thousand to one data for…