Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

111–120 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#111

Post by ivankelly »

sh00s wrote: post #110
ivankelly wrote: post #109

Earlier quoted context omitted.

Does it block reads until sealing is complete? How many nodes in the nodeset have to respond before sealing is complete? [NodeSet] - [ReplicationFactor] + 1?

Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority. For a simple case where placement of data is location-agnostic, indeed the definition of f-majority is n - r + 1, where n is the nodeset size, and r is the replication factor. However, if your replication property…

> Yes, reads are not released (i.e. are blocked) until sealing is complete.

Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read.

> However, if your replication property, is say, "place 3 copies across 3 racks", then the definition of f-majority becomes more complicated - e.g. having all nodes in the nodeset respond minus two racks will also satisfy it.

Sure, the aim being that no write can be successfully acknowledged by enough replicas to complete the write.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#112

Post by sh00s »

sh00s wrote: post #110
ivankelly wrote: post #109

Earlier quoted context omitted.

Does it block reads until sealing is complete? How many nodes in the nodeset have to respond before sealing is complete? [NodeSet] - [ReplicationFactor] + 1?

Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority. For a simple case where placement of data is location-agnostic, indeed the definition of f-majority is n - r + 1, where n is the nodeset size, and r is the replication factor. However, if your replication property…

> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read.

Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#113

Post by ivankelly »

sh00s wrote: post #112
sh00s wrote: post #110

Earlier quoted context omitted.

Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority. For a simple case where placement of data is location-agnostic, indeed the definition of f-majority is n - r + 1, where n is the nodeset size, and r is the replication factor. However, if your replication property…

> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read. Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out…

> Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee.

But given that reads are blocked on all sequencers before the current one, this should still provide total order atomic broadcast, unless a single client can connect to a sequencer with a lower epoch than one it has already seen.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#114

Post by sh00s »

ivankelly wrote: post #113
sh00s wrote: post #112

Earlier quoted context omitted.

> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read. Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out…

> Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee. But given that reads are blocked on all sequencers before the current one, this should still provide total order atomic broadcast, unless a sing…

LogDevice clients do notify sequencers if they have seen newer epochs, which would cause a sequencer reactivation, which indeed resolves the issue within the context of a single client.

However, there can still be reordering in the context of a wider system. E.g. if client A sends a write (w1) to sequencer in epoch X, which gets replicated and acknowledged, and after that client B sends a write (w2) to sequencer in epoch (X-1) which gets replicated and acknowledged (because epoch X-1 is not sealed), then readers eventually will see w2 before w1. If writes in epoch X weren't accepted before the sealing of the epoch (X-1) had completed, this reordering would be impossible, however as a result write availability would suffer.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#115

Post by sh00s »

ivankelly wrote: post #113
sh00s wrote: post #112

Earlier quoted context omitted.

> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read. Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out…

> Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee. But given that reads are blocked on all sequencers before the current one, this should still provide total order atomic broadcast, unless a sing…

Ah, I think you may be talking about the repeatable reads property? All readers in LogDevice are guaranteed to see the same records in the same order (aside from trimmed data).

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#116

Post by ivankelly »

sh00s wrote: post #114
ivankelly wrote: post #113

Earlier quoted context omitted.

> Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee. But given that reads are blocked on all sequencers before the current one, this should still provide total order atomic broadcast, unless a sing…

LogDevice clients do notify sequencers if they have seen newer epochs, which would cause a sequencer reactivation, which indeed resolves the issue within the context of a single client. However, there can still be reordering in the context of a wider system. E.g. if client A sends a write (w1) to sequencer in epoch X, which gets replicated and acknowledged, and after that client B sends a write (w2) to sequencer in e…

Ok, but for this to be problematic, readers would need to have some other mechanism to know that w1 did actually take place before w2. So FIFO instead of total order.

Anyhow, thanks for answering my questions. Very interesting system.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#117

Post by ivankelly »

sh00s wrote: post #115
ivankelly wrote: post #113

Earlier quoted context omitted.

> Consistency in a more general sense than just read-modify-write consistency. If you have sequencers active in several epochs at the same time accepting writes, the records may end up being written out of order, and there would be a breakage of the total ordering guarantee. But given that reads are blocked on all sequencers before the current one, this should still provide total order atomic broadcast, unless a sing…

Ah, I think you may be talking about the repeatable reads property? All readers in LogDevice are guaranteed to see the same records in the same order (aside from trimmed data).

What I was wondering really, was whether LogDevice provides total order atomic broadcast, and as such whether it solves concensus. It appears it does (or rather, it daisychains on the concensus provided by zookeeper and uses it's own fencing mechanism, similar to what bookkeeper/Pulsar does).

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#118

Post by windexh8er »

Serow225 wrote: post #70
EdwardDiego wrote: post #57

Earlier quoted context omitted.

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)

Gravwell.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#119

Post by dang »

Annatar wrote: post #108
javiermaestro wrote: post #99

Earlier quoted context omitted.

A few things. First, can't you run Linux inside a Solaris Zone? I don't know much about Solaris stuff (although I do like it very much, I grew up mostly with Linux, which you so much despise, and I'm not too familiar with other Unixes). So... I think you could probably run Logdevice if you really wanted. Then, here's my two cents. When engaging in conversation and civil dialog, please try to avoid being so dismissive…

"You come across as abrasive and entitled." Does it appear to you that I care? I am abrasive. And I am entitled. I'm also very proud that I run an illumos based operating system instead of a GNU/Linux one. Anything else I can clarify for you? "I think you could probably run Logdevice if you really wanted." Nowhere did I write that I actually care about logdevice. What I care about is that these kids learn UNIX: daemo…

Please follow the site guidelines.

https://news.ycombinator.com/newsguidelines.html

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#120

Post by Annatar »

dang wrote: post #119
Annatar wrote: post #108

Earlier quoted context omitted.

"You come across as abrasive and entitled." Does it appear to you that I care? I am abrasive. And I am entitled. I'm also very proud that I run an illumos based operating system instead of a GNU/Linux one. Anything else I can clarify for you? "I think you could probably run Logdevice if you really wanted." Nowhere did I write that I actually care about logdevice. What I care about is that these kids learn UNIX: daemo…

Please follow the site guidelines. https://news.ycombinator.com/newsguidelines.html

Which guideline do you believe I did not follow? (I hope you write back face to face, because I would tell him the same thing in person and then some, and enjoy every microsecond of it.)
Post reply on HN