Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

71–80 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#71

Post by jacobr1 »

Serow225 wrote: post #70
EdwardDiego wrote: post #57

Earlier quoted context omitted.

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)

AWS Kinesis + s3

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#72

Post by tveita »

the_duke wrote: post #38
AhmedSoliman wrote: post #4

Happy to finally see LogDevice open. We have been working on this for years now.

Can you give an overview over the difference to eg Apache Kafka? It seems very similar.

From what I can see this doesn't have built-in consumer balancing and offset storage, like Kafka does. It also lacks more exotic Kafka features like topic compaction and exactly-once processing.

In Kafka bulk reading is very cheap, the broker basically just calls sendfile() to send a file segment with compressed message chunks. On the other hand only the leader of a partition can serve requests, so you are often limited by bandwidth. It looks like LogDevice has to do a bit more work server side, but may be able to read from all servers with a replica.

Kafka stores more metadata in the record wrapper, like client and server timestamps and partition key.

There are client libraries for C++ and Python.

Operationally they look similar - both require a Zookeeper cluster, and both require assigning permanent ids to nodes.

It would be interesting to see some benchmarks comparing LogDevice with Kafka and Pulsar. That said, I suspect from the lack of buzz around Pulsar that Kafka isn't a performance bottleneck for most people using it.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#73

Post by thrusong »

martincmartin wrote: post #26
tryptophan wrote: post #6

What benefit to facebook is there from open sourcing technology they have developed?

Facebook's competitive advantage doesn't come from having the best reliable streaming data store at scale, or from its software in general. Even if MySpace, Friendster or Google + got their hands on the whole software stack & started running it, people would stick with Facebook. So there's no cost to open sourcing. The benefit comes from being known as technically innovative in general, and for recruiting, being know…

If that's true, why haven't they open sourced Haystack? Clearly they're holding onto it due to competitive advantage.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#74

Post by simonrobb »

Serow225 wrote: post #70
EdwardDiego wrote: post #57

Earlier quoted context omitted.

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)

OKLog, Humio, and Splunk are all worth checking out.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#75

Post by majidazimi »

LgWoodenBadger wrote: post #64
majidazimi wrote: post #48

Earlier quoted context omitted.

True, but Kafka has two very annoying features built into it: - There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes. - Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distribu…

I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

No it doens't, since a single partition is stored sequentially on one disk which limits the consumers to bandwidth of single disk (say c1 reads beginning of the partition and c2 end of the partition). But in the case of Pulsar c1 is most probably connected to a different node than c2.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#77

Post by Serow225 »

jacobr1 wrote: post #71
Serow225 wrote: post #70

Earlier quoted context omitted.

What's a good distributed log for 10-dev sized companies? :)

AWS Kinesis + s3

Thanks! I'm guessing you're referring to Kinesis Streams? Is there an OOB solution to persist the records past the default 168hours, or is this something that you have to build out yourself following some pattern?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#78

Post by martincmartin »

thrusong wrote: post #73
martincmartin wrote: post #26

Earlier quoted context omitted.

Facebook's competitive advantage doesn't come from having the best reliable streaming data store at scale, or from its software in general. Even if MySpace, Friendster or Google + got their hands on the whole software stack & started running it, people would stick with Facebook. So there's no cost to open sourcing. The benefit comes from being known as technically innovative in general, and for recruiting, being know…

If that's true, why haven't they open sourced Haystack? Clearly they're holding onto it due to competitive advantage.

I don't know, but my guess is nobody associated with it wants to put their other work on hold to make it happen. From my limited experience, nobody pushes you, and nobody blocks you. So it depends a lot on the motivations of the engineers on the project.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#79

Post by atombender »

simonrobb wrote: post #74
Serow225 wrote: post #70

Earlier quoted context omitted.

What's a good distributed log for 10-dev sized companies? :)

OKLog, Humio, and Splunk are all worth checking out.

OKLog has been abandoned by the author (the project is now read only on GitHub).

Humio is not self-hosted or open source, so not really a fair comparison. It also seems targeted towards operational logs, i.e. system logging, traffic logging, auditing. Not things like data pipelines. Kafka and friends can be used for that kind of log, but they are more like databases; they use the term "log" in the sense of sequential and append-only.

Same goes for Splunk, which does have a self-hosted version, but is extremely expensive, last I checked. The SaaS version is also extremely expensive.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#80

Post by sh00s »

jMyles wrote: post #63
cedricvg wrote: post #59

Earlier quoted context omitted.

LogDevice uses SSL for authentication. This can be enabled for both clients and servers [1]. [1] https://logdevice.io/docs/Settings.html#security

That's not what I mean though. What if I have a cluster with devices I don't trust, but I want to let them emit logs if they conform to a particular protocol. Like, will this thing check signatures for me and such? Since it doesn't say anything about trustlessness, I assume that it assumes that all nodes are trusted.

LogDevice is payload-agnostic and doesn't inspect the value of the binary blobs it stores. If your writer is allowed to write according to ACLs, LogDevice will happily take writes from it regardless of their content (upto the max payload size). Verification of payload content should be done on a layer above LogDevice - either before taking the write or when reading.
Post reply on HN