Scribe: Transporting petabytes per hour via a distributed, buffered queueing
1–10 of 44 posts
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#2Keep in mind the network cost of petabytes per hour cross continent.
Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#3Keep in mind the network cost of petabytes per hour cross continent. Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
Sounds like a startup idea.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#4Keep in mind the network cost of petabytes per hour cross continent. Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
Never underestimate the bandwidth of a cargo ship full of SSDs.
(I'm paraphrasing an old, old joke.)
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#5Naming is hard! Facebook used to have a _different_ Scribe (https://en.wikipedia.org/wiki/Scribe_(log_server)).
We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#6Naming is hard! Facebook used to have a _different_ Scribe ( https://en.wikipedia.org/wiki/Scribe_(log_server) ). We used it at a company I worked for, but it had long-since been deprecated, so I was confused when I saw this Scribe.
This is the same scribe.
Facebook closed-sourced it because it was too hard to maintain an open version and a version that addresses Facebook's needs.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#7Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#8Keep in mind the network cost of petabytes per hour cross continent. Those of us who don't own our own cross-ocean fiber can't afford to design systems like this.
Never underestimate the bandwidth of a cargo ship full of SSDs. (I'm paraphrasing an old, old joke.)
Fairly high latency though I guess.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#9braggy PR is misleading: the 25GB/s coming from CERN is after they filter the data down from 600TB/s because there are no commercial systems that can capture data at higher rates.
Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing
#10How does this compare to a robust Splunk infrastructure?