Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

41–50 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#41

Post by StreamBright »

manigandham wrote: post #16
ashu wrote: post #13

Earlier quoted context omitted.

How does LogDevice differ from Kafka?

Kafka brokers handle both the computation (partition/topic management, sequencing, assignments, etc) and storage together. This coupling creates scaling and operational challenges which LogDevice removes by separating the layers. Storage nodes can be as simple as object stores (but optimized for appending files) and use multiple non-deterministic locations for a given piece of data to randomize placement. They read,…

Amazing! We had lots of operational issues because of the coupling you mentioned.

One question though: will Presto support querying from LogDevice directly? :)

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#42

Post by AhmedSoliman »

adev_ wrote: post #37

Thank to Open Source that, it looks a great project. Could a LogDevice give a bit of informations about the scale they use that at facebook ? - How many record this thing can injest per day ? - Any limitations on the maximum number of storage nodes ? - What would be your maximum and advise size of record for a production usage ? - ZooKeeper seems to be the center point used as epoch provider. Did you encounter any sc…

I cannot give you exact numbers, but here are some information that might be useful: - LogDevice ingests over 1TB/s of uncompressed data at Facebook. This already has been highlighted in last year's talk in @Scale conference. - The maximum limit as defined by default in the code for the number of storage nodes in a cluster is 512. However, you can use --max-nodes to change that. There is no theoretical limit there. Each LogDevice storage daemon can handle multiple physical disks (we call them shards). So, If you have 15 disks per box, 512 servers. That's 7680 total disks in a single cluster. - The maximum record size is 32MB. However, in practice, payloads are usually much smaller. - Zookeeper is not (currently) a scaling limitation as we don't connect to zookeeper from Clients (as long as you are sourcing the config file from filesystem and not using zookeeper for that as well).

Hope that helps.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#43

Post by AhmedSoliman »

mmcclellan wrote: post #3

I had just stumbled across https://github.com/facebookincubator/python-nubia and am anxious to try it out. Was wondering about the internal project it was factored out from. This appears to be it.

Correct. LDShell in logdevice was the starting point of python-nubia.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#44

Post by AhmedSoliman »

the_duke wrote: post #38
AhmedSoliman wrote: post #4

Happy to finally see LogDevice open. We have been working on this for years now.

Can you give an overview over the difference to eg Apache Kafka? It seems very similar.

It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works.

In terms of function. LogDevice is similar to the core of Apache Kafka.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#45

Post by AhmedSoliman »

sandstrom wrote: post #32

Very interesting! I like the idea of decoupling compute from storage for streaming/log data. I wonder if it would be easy to make it run under Consul, instead of ZooKeeper.

We use Zookeeper primarily for the EpochStore. This is the abstraction that you can you use if you want to replace Zookeeper. It shouldn't be that hard as long as Consul offers the same guarantees as zookeeper.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#47

Post by majidazimi »

External logging service is my favorite way of doing replication. It provides nice features. Specifically:

- Cross vendor replication which makes migration much easier.

- No dependency on vendor provided replication protocols.

- Ability to use in-app databases such RocksDB, SQLite, ...

- Upgrading DB nodes becomes way easier since they are totally separated from each other.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#48

Post by majidazimi »

AhmedSoliman wrote: post #44
the_duke wrote: post #38

Earlier quoted context omitted.

Can you give an overview over the difference to eg Apache Kafka? It seems very similar.

It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works. In terms of function. LogDevice is similar to the core of Apache Kafka.

True, but Kafka has two very annoying features built into it:

- There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes.

- Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distributed among broker nodes which helps a lot when there are many readers.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#49

Post by cedricvg »

akavel wrote: post #30

Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#50

Post by Annatar »

"bin/logdeviced"

All daemons and system administration utilities belong into sbin, because bin is for end-user applications.

Historically, the "s" in sbin meant something else, but it always contained applications and scripts only root could run.

When I see these examples, it's depressing to see just how much understanding of UNIX is missing.

Post reply on HN