Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scribe: Transporting petabytes per hour via a distributed, buffered queueing

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#31

Post by adev_ »

gnufx wrote: post #20
dekhn wrote: post #18

Earlier quoted context omitted.

it's pretty simple. the physical data acquisition devices (ATLAS is an example) collect data at rates in the 100s of terabytes/sec https://home.cern/science/computing/processing-what-record ) No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They…

I know how data collection works, but it sounded as if 25GB/s was regarded as high compared with filesystems you can buy. Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.

> Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre

Data model at CERN does not match the one of a supercomputer. CERN data are not processed locally but distributed and spread to ~100 of participating institute in the experiment.

Moreover, "personal opinion", GPFS is crap. It's an old relic from the 90s that has so many quirk and problem of design that it would deserves an entire conference on it. Plus the fact it's proprietary and expensive.

The only reason that make GPFS still alive is that for a long time, the only alternative was Lustre, and Lustre is even worst.

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#32

Post by bradhe »

javiermaestro wrote: post #26
bradhe wrote: post #22

So, if I'm reading this correctly, 2.5GB/s of log data being generated? If we assume (aggressively) that they have 5mil machines in their infrastructure, doesn't that mean that each machine would have to be generating 500kB/s of log data? Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.

The number in the article is 2.5 TB/s, not GB/s :) (disclaimer: I work in Scribe)

Right—sorry. But point still stands. Under what circumstances was that much data being generated from (what I’m assuming is) normal logging?

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#34

Post by javiermaestro »

bradhe wrote: post #32
javiermaestro wrote: post #26

Earlier quoted context omitted.

The number in the article is 2.5 TB/s, not GB/s :) (disclaimer: I work in Scribe)

Right—sorry. But point still stands. Under what circumstances was that much data being generated from (what I’m assuming is) normal logging?

I'm not following. I understood from your first comment that you think the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data").

In any case, the data is "whatever needs to be logged".

And it's not "server logs", which is what I'm interpreting from your comment. Scribe transports most data at Facebook to be processed by real-time systems (e.g. Puma, Scuba) and also "batch systems" (data warehouse). So, it's quite a lot, being "the ingestion pipe" for Facebook.

Does this answer your question? :-?

Puma: https://research.fb.com/publications/realtime-data-processin...

Scuba: https://research.fb.com/publications/scuba-diving-into-data-...

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#35

Post by mkasu »

packetslave wrote: post #27
dekhn wrote: post #9

braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.

it's adorable how you call it "braggy PR" when almost every major technology company these days (FB, Google, Amazon, Uber, Pinterest, etc., pretty much everybody except Apple) has an engineering blog where they share possibly-interesting work they've done.

Apple also has a blog where they discuss (some of) their machine learning results[1].

1: https://machinelearning.apple.com/

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#37

Post by thetrooperer »

pvlak wrote: post #36

why can't Producer/Scribed write directly to LogDevice storage. Any reasons for routing through WriteService.

That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the producer hosts.

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#38

Post by dekhn »

throwaway_bad wrote: post #28
breck wrote: post #19

Earlier quoted context omitted.

This is a good point! Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)

Are you trying to quantify the "information" by the size of the DNA? I think this is a pretty meaningless number to multiply since most of the DNA will be exact copies and DNA alone doesn't capture all the information about a cell. OTOH the amount of "information" needed to perfectly simulate a cell is probably unbounded. Just a corollary of the fact that we currently don't know how to perfectly simulate reality. Eve…

It's worth reading the prior literature: Markus Covert has gotten pretty good at predicting quantitative phenotypes using whole cell simulations (with very limited cell representations, basically just feature matrices).

https://www.cell.com/abstract/S0092-8674(12)00776-3

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#39

Post by dekhn »

gnufx wrote: post #20
dekhn wrote: post #18

Earlier quoted context omitted.

it's pretty simple. the physical data acquisition devices (ATLAS is an example) collect data at rates in the 100s of terabytes/sec https://home.cern/science/computing/processing-what-record ) No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They…

I know how data collection works, but it sounded as if 25GB/s was regarded as high compared with filesystems you can buy. Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.

lustre is crap.

Every single supercomputer meeting I've been to (I've been part of the community for years, they often invite me to their meetings to give an industry perspective), people are just continuously complaining about the filesystems, and it's GPFS and Lustre at the top of the list.

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#40

Post by bradhe »

javiermaestro wrote: post #34
bradhe wrote: post #32

Earlier quoted context omitted.

Right—sorry. But point still stands. Under what circumstances was that much data being generated from (what I’m assuming is) normal logging?

I'm not following. I understood from your first comment that you think the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data"). In any case, the data is "whatever needs to be logged". And it's not "server logs", which is what I'm interpreting from your comment. Scribe transports most data at Facebook to be processed by real-time systems (e.g. Puma, Scuba) and also "ba…

> So, it's quite a lot, being "the ingestion pipe" for Facebook.

I see. I walked away from the article with the impression that it was meant to be a log aggregation service a la flume, splunk, or logstash.

> the amount of data is low ("underwhelming") and from your last comment that it's a lot ("that much data").

I was remarking on the numbers in regard to generation, not consumption. Based on the article, my estimate is pointing out that generating 2.5TB/s of transactional logs and telemetry data using "millions" of machines would be technically possible but not reasonably practical...and thus likely not real ;). But, you corrected my understanding: That number isn't based on a different use case.

Post reply on HN