Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

61–70 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#61

Post by akavel »

cedricvg wrote: post #49
akavel wrote: post #30

Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.

Is some form of compression enabled by default, without having to tweak options?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#62

Post by are595 »

SirMonkey wrote: post #55
akavel wrote: post #30

Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

have you had a look at https://github.com/oklog/oklog https://www.youtube.com/watch?v=gWWK2eyZ-sc I think it's fairly simple and might be enough. Can't comment on storage requirements thou.

Dang, I was looking at oklog earlier, but looks like it is archived now...

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#63

Post by jMyles »

cedricvg wrote: post #59
jMyles wrote: post #58

I don't see anything about trust requirements or verification. Does LogDevice assume that all devices in my cluster are trusted?

LogDevice uses SSL for authentication. This can be enabled for both clients and servers [1]. [1] https://logdevice.io/docs/Settings.html#security

That's not what I mean though. What if I have a cluster with devices I don't trust, but I want to let them emit logs if they conform to a particular protocol. Like, will this thing check signatures for me and such?

Since it doesn't say anything about trustlessness, I assume that it assumes that all nodes are trusted.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#64

Post by LgWoodenBadger »

majidazimi wrote: post #48
AhmedSoliman wrote: post #44

Earlier quoted context omitted.

It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works. In terms of function. LogDevice is similar to the core of Apache Kafka.

True, but Kafka has two very annoying features built into it: - There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes. - Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distribu…

I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#65

Post by [deleted] »

tinco wrote: post #35
fullmetaleng wrote: post #33

Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - https://twitter.com/martinkl/status/1039938408393662465

Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. Do people just point their system journal at Kafka and wait for something to break? At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might…

[deleted]

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#66

Post by thinkersilver »

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar

The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writes at scale but this is pure speculation.

I don't know. If I understand why anyone might use this I'd contribute to building language bindings for the APIs.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#67

Post by lexs »

jMyles wrote: post #63
cedricvg wrote: post #59

Earlier quoted context omitted.

LogDevice uses SSL for authentication. This can be enabled for both clients and servers [1]. [1] https://logdevice.io/docs/Settings.html#security

That's not what I mean though. What if I have a cluster with devices I don't trust, but I want to let them emit logs if they conform to a particular protocol. Like, will this thing check signatures for me and such? Since it doesn't say anything about trustlessness, I assume that it assumes that all nodes are trusted.

LogDevice is crash fault tolerant not byzantine fault tolerant if that's what you're asking. This fault tolerance is in regards to where the logs are placed not who's emitting them though. If you want to analyse logs for inconsistencies or attack patterns you should look into something like SEAMS/REAMS, it's completely out of scope for LogDevice.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#68

Post by thinkersilver »

AhmedSoliman wrote: post #4

Happy to finally see LogDevice open. We have been working on this for years now.

Hi Ahmed,

I imagine you looked at other solutions before starting this. A distributed log is a fairly simple idea to understand (hard to implement) but what pain point is being solved?

Seeing that it is written in C/C++ - would it be that logdevice is optimised purely for speed and responsiveness?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#69

Post by otterley »

thinkersilver wrote: post #66

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writ…

> Is it because Kafka doesn't scale to millions of logs on a single cluster

I doubt that's it, since Kafka can certainly do that.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#70

Post by Serow225 »

EdwardDiego wrote: post #57
tinco wrote: post #35

Earlier quoted context omitted.

Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. Do people just point their system journal at Kafka and wait for something to break? At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might…

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)
Post reply on HN