Scribe: Transporting petabytes per hour via a distributed, buffered queueing • Hacker News Board

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#41

Post by pvlak » Wed, Oct 09, 2019, 2:47 AM UTC

thetrooperer wrote: post #37

pvlak wrote: post #36

why can't Producer/Scribed write directly to LogDevice storage. Any reasons for routing through WriteService.

That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the prod…

Will the WriteService(Aggregator) make sense for environments having thousands of machines(not in millions) and they are all within the DataCenter. In our company, we are moving away from this design of having aggregators, to directly writing to Storage whereever possible, as it reduces the message loss.

On the volume of metadata held by Producers, will there be any significant difference between holding WriteService & LogDevice meta.

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#42

Post by thetrooperer » Wed, Oct 09, 2019, 2:53 PM UTC

pvlak wrote: post #41

thetrooperer wrote: post #37

Earlier quoted context omitted.

That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the prod…

Will the WriteService(Aggregator) make sense for environments having thousands of machines(not in millions) and they are all within the DataCenter. In our company, we are moving away from this design of having aggregators, to directly writing to Storage whereever possible, as it reduces the message loss. On the volume of metadata held by Producers, will there be any significant difference between holding WriteService…

The devil is in the details probably, but if you have a single datacenter and all writes are coming from thousands of machines ("edge"), yes, it may make more sense to set up a single LogDevice / Kafka cluster and have all the edge hosts write to it directly.

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#43

Post by derefr » Fri, Oct 11, 2019, 10:59 PM UTC

dekhn wrote: post #39

gnufx wrote: post #20

Earlier quoted context omitted.

I know how data collection works, but it sounded as if 25GB/s was regarded as high compared with filesystems you can buy. Obviously some people do want GPFS, if they can afford it, but Cori uses Lustre. I don't mean to claim that either is ideal for streaming high rate event data, of course.

lustre is crap. Every single supercomputer meeting I've been to (I've been part of the community for years, they often invite me to their meetings to give an industry perspective), people are just continuously complaining about the filesystems, and it's GPFS and Lustre at the top of the list.

What filesystems would they like to be using?

Re: Scribe: Transporting petabytes per hour via a distributed, buffered queueing

#44

Post by derefr » Fri, Oct 11, 2019, 11:01 PM UTC

trhway wrote: post #21

noir_lord wrote: post #8

Earlier quoted context omitted.

Fairly high latency though I guess.

using jet for trans ocean data delivery gives you several hours latency - acceptable for the logs - at the cost on the scale like $0.1-0.3/TB (really depends on the napkin used for the estimations)

Big capital costs in setting up the number of parallel writers required on one end and readers required on the other, though. And presumably human "IT teamster" labor, hooking and unhooking drives.