11–20 of 44 posts
Naming is hard! Facebook used to have a _different_ Scribe ( https://en.wikipedia.org/wiki/Scribe_(log_server) ). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
Naming is hard! Facebook used to have a _different_ Scribe ( https://en.wikipedia.org/wiki/Scribe_(log_server) ). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
How does this compare to a robust Splunk infrastructure?
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
Naming is hard! Facebook used to have a _different_ Scribe ( https://en.wikipedia.org/wiki/Scribe_(log_server) ). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
This is the same scribe. Facebook closed-sourced it because it was too hard to maintain an open version and a version that addresses Facebook's needs.
Is it normal for these internal system to not implement any kind of access control? From the post it seems every reader can access every stream?
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
I'm not sure what that means, but the Cori filesystem is rated at 700GB/s and Summit's 2.5TB/s. See https://docs.nersc.gov/filesystems/cori-scratch/ and https://www.olcf.ornl.gov/olcf-resources/compute-systems/sum...
No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They tune the thresholds to match what consumer hardware is capable of.
With regard to supercomputer filesystems: nobody wants to use GPFS. CERN's EOS sustained (theoretical) 3.3TB/sec in Apr 2015, so it's not like they're uncompetitive with the largest supercomputer...
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)
Earlier quoted context omitted.
I'm not sure what that means, but the Cori filesystem is rated at 700GB/s and Summit's 2.5TB/s. See https://docs.nersc.gov/filesystems/cori-scratch/ and https://www.olcf.ornl.gov/olcf-resources/compute-systems/sum...
it's pretty simple. the physical data acquisition devices (ATLAS is an example) collect data at rates in the 100s of terabytes/sec https://home.cern/science/computing/processing-what-record ) No storage system can store that data (and most of it is not useful) so they have a series of hardware triggers and buffers that reduce the data down to roughly what modern (general purpose) hardware is capable of handling. They…
Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.