21–30 of 44 posts
Earlier quoted context omitted.
Never underestimate the bandwidth of a cargo ship full of SSDs. (I'm paraphrasing an old, old joke.)
Fairly high latency though I guess.
Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.
Is it normal for these internal system to not implement any kind of access control? From the post it seems every reader can access every stream?
The article just focuses on certain areas of the system and doesn't go into the security and privacy parts, that's all.
(I work in Scribe)
Earlier quoted context omitted.
This is the same scribe. Facebook closed-sourced it because it was too hard to maintain an open version and a version that addresses Facebook's needs.
I wonder why couldn't they just make the open source version address their needs...
Maybe one day we'll have a version available. In any case, one of the larger parts of the system (LogDevice) is open source :)
(disclaimer: I work in Scribe)
LogDevice: https://engineering.fb.com/core-data/open-sourcing-logdevice...
Naming is hard! Facebook used to have a _different_ Scribe ( https://en.wikipedia.org/wiki/Scribe_(log_server) ). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
Scribe is also an enterprise integration/ETL tool. For _many_ years before FB existed.
https://web.archive.org/web/20120301000000*/http://www.scrib...
The initial stuff seems to not be so related, but the current description of what they do seems much much closer to what Facebook's Scribe does today :)
Naming is hard! :D
So, if I'm reading this correctly, 2.5GB/s of log data being generated? If we assume (aggressively) that they have 5mil machines in their infrastructure, doesn't that mean that each machine would have to be generating 500kB/s of log data? Despite that, I find the claims to be underwhelming. So your system can process massive amounts of data by scaling massively horizontally...neat.
(disclaimer: I work in Scribe)
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
This is a good point! Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)
OTOH the amount of "information" needed to perfectly simulate a cell is probably unbounded. Just a corollary of the fact that we currently don't know how to perfectly simulate reality. Even a single "real" number can take up infinite space.
Earlier quoted context omitted.
This is a good point! Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)
Are you trying to quantify the "information" by the size of the DNA? I think this is a pretty meaningless number to multiply since most of the DNA will be exact copies and DNA alone doesn't capture all the information about a cell. OTOH the amount of "information" needed to perfectly simulate a cell is probably unbounded. Just a corollary of the fact that we currently don't know how to perfectly simulate reality. Eve…
This is a very good point. The 'information' in a cell isn't the base pairs in its DNA, but all the atoms that make up the whole cell. And then each atom encapsulates properties such as position, velocity, charge, van der Waals radius etc.
However this considers atoms with classical mechanics. In a quantum mechanical representation it would be very different again and you can start asking really hairy questions about whether information can be created or destroyed.
Earlier quoted context omitted.
This is a good point! Just for fun, for more perspective on big data, a human body generates around 1-10M new cells per second, and a cell contains about 10-100GB of information. So a single human is generating 1-100PB/s of data just in the new cells! (Give or take a few OOM)
Are you trying to quantify the "information" by the size of the DNA? I think this is a pretty meaningless number to multiply since most of the DNA will be exact copies and DNA alone doesn't capture all the information about a cell. OTOH the amount of "information" needed to perfectly simulate a cell is probably unbounded. Just a corollary of the fact that we currently don't know how to perfectly simulate reality. Eve…