Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

81–90 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#81

Post by atombender »

Serow225 wrote: post #70
EdwardDiego wrote: post #57

Earlier quoted context omitted.

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)

If you're willing to go cloud, then Google's BigQuery is a very good fit for sequential data. It's fully queryable, unlike most log-oriented databases, and it's extremely cheap compared to competing offerings (e.g. AWS RedShift).

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#82

Post by sh00s »

thinkersilver wrote: post #66

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writ…

Some strengths of LogDevice include:

- It's designed to work with a large number of logs (roughly equivalent to partitions in Kafka), hundreds of thousands per cluster is common.

- Sequencer failover is very quick, typical failover time when a sequencer node fails is less than a second.

- It supports location awareness and can place data according to replication constraints specified (e.g. replicate it in 3 copies across 2 different regions and 3 racks).

- Because of non-deterministic data placement, it is very resilient to failures in terms of write availability.

- If a node/shard fails, it detects the failure and rebuilds the data that was replicated to failed nodes/shards automatically

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#83

Post by sh00s »

akavel wrote: post #61
cedricvg wrote: post #49

Earlier quoted context omitted.

These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.

Is some form of compression enabled by default, without having to tweak options?

No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#84

Post by sh00s »

majidazimi wrote: post #48
AhmedSoliman wrote: post #44

Earlier quoted context omitted.

It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works. In terms of function. LogDevice is similar to the core of Apache Kafka.

True, but Kafka has two very annoying features built into it: - There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes. - Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distribu…

LogDevice has many-to-many rebuilding as well, and typically data for a log (similar to partition in Kafka) is spread relatively uniformly over the (potentially large, much bigger than replication factor) set of shards that hold data for that log.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#85

Post by sh00s »

LgWoodenBadger wrote: post #64
majidazimi wrote: post #48

Earlier quoted context omitted.

True, but Kafka has two very annoying features built into it: - There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes. - Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distribu…

I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

splitting data into partitions would mean there's no total order on that data anymore, right?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#86

Post by adrienconrath »

majidazimi wrote: post #75
LgWoodenBadger wrote: post #64

Earlier quoted context omitted.

I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

No it doens't, since a single partition is stored sequentially on one disk which limits the consumers to bandwidth of single disk (say c1 reads beginning of the partition and c2 end of the partition). But in the case of Pulsar c1 is most probably connected to a different node than c2.

LogDevice has this concept of "node set", which is the set of storage nodes that can be selected by the sequencer as recipients for a record or a block of records. A typical node set size is around 20-30 in our deployments. Each storage node in the node set contains a subset of the records (or blocks of records) of the log, we call that subset a log strand. The amount of IO capacity available to append records to a log or read records from a log scales with the size of the node set.

All of this is done while preserving the total ordering guarantee thanks the separation of sequencing and storage.

The operator could for example set a bigger node set size for logs that are known to have multiple consumers and require more IO capacity.

At facebook, we have use cases where a single consumer will need to replay a backlog of records in a log, sometimes hours or days worth of data to rebuild its state. We call this a backfill. Node sets allow the IO to be spread across multiple disks which improves backfill speed and helps reduce hotspots.

-- Adrien from the LogDevice team.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#87

Post by LgWoodenBadger »

sh00s wrote: post #85
LgWoodenBadger wrote: post #64

Earlier quoted context omitted.

I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

splitting data into partitions would mean there's no total order on that data anymore, right?

Kafka only guarantees a total order within a given partition, not across them.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#88

Post by adrienconrath »

sh00s wrote: post #82
thinkersilver wrote: post #66

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writ…

Some strengths of LogDevice include: - It's designed to work with a large number of logs (roughly equivalent to partitions in Kafka), hundreds of thousands per cluster is common. - Sequencer failover is very quick, typical failover time when a sequencer node fails is less than a second. - It supports location awareness and can place data according to replication constraints specified (e.g. replicate it in 3 copies ac…

> Because of non-deterministic data placement, it is very resilient to failures in terms of write availability.

I am happy to expand more on this point.

We have this concept of "node set" of a log which is the set of storage nodes available to receive record copies sent by the sequencer. It is typically made of 20-30 nodes in typical deployments at Facebook. Write availability is maintained as long as enough storage nodes in the node set are available to accept copies. When storage node failures are detected, the sequencer can just exclude these nodes from the list of potential recipients for new records. It does not need to update a view that needs to be synchronized with readers, which is a heavy-weight operation. This model allows preserving high write availability even if many nodes in the node set are unhealthy.

Additionally, this record copy placement flexibility allows the sequencer to quickly route around latency spikes on individual storage nodes, which helps guarantee low append latency.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#89

Post by mhotchen »

Serow225 wrote: post #77
jacobr1 wrote: post #71

Earlier quoted context omitted.

AWS Kinesis + s3

Thanks! I'm guessing you're referring to Kinesis Streams? Is there an OOB solution to persist the records past the default 168hours, or is this something that you have to build out yourself following some pattern?

You can configure it to output to S3 and the mechanism for that is easy to configure and has fault tolerance.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#90

Post by manigandham »

otterley wrote: post #69
thinkersilver wrote: post #66

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writ…

> Is it because Kafka doesn't scale to millions of logs on a single cluster I doubt that's it, since Kafka can certainly do that.

Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.
Post reply on HN