51–60 of 123 posts
"bin/logdeviced" All daemons and system administration utilities belong into sbin, because bin is for end-user applications. Historically, the "s" in sbin meant something else, but it always contained applications and scripts only root could run. When I see these examples, it's depressing to see just how much understanding of UNIX is missing.
Earlier quoted context omitted.
Scribe is the Facebook-internal Kafka equivalent. LogDevice is the storage layer used by Scribe. Scribe isn’t the only place where LogDevice is used though — Facebook has documented using it for TAO as well (as part of the secondary indices)
I don't believe Scribe and Kafka are equivalent. Isn't Scribe at-most-once? Unless we're talking about two different projects named Scribe, which is certainly possible.
At any rate, I used “equivalent” here to mean that, while different trade-offs have been made, it has the sort of users building the same sort of applications on the same sort of abstractions — it plays the same role, for all intents and purposes.
Earlier quoted context omitted.
I don't believe Scribe and Kafka are equivalent. Isn't Scribe at-most-once? Unless we're talking about two different projects named Scribe, which is certainly possible.
Scribe doesn't really make decisions about that, it doesn't store checkpoints for readers. Readers are commonly more-than-once.
It kind of does, on the write pipeline. Tailers vary, but scribed controls the semantics of how your message gets delivered to LogDevice.
What benefit to facebook is there from open sourcing technology they have developed?
Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...
I think it's fairly simple and might be enough. Can't comment on storage requirements thou.
Earlier quoted context omitted.
Can you give an overview over the difference to eg Apache Kafka? It seems very similar.
It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works. In terms of function. LogDevice is similar to the core of Apache Kafka.
Curious where logdevice would be a bad decision..
Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - https://twitter.com/martinkl/status/1039938408393662465
Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. Do people just point their system journal at Kafka and wait for something to break? At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might…
Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads.
LogDevice looks to be very similar for most use cases to Kafka, hell, they even use RocksDB, which is used by stateful operations in Kafka Streaming, and of course, Zookeeper.
Where it differs is that it looks like it was designed for you to be able to work against a single "cluster" that could well be running across multiple data-centres. Which is very much a Facebook problem to solve.
So yeah, Kafka was a distributed log built for LinkedIn size problems, LogDevice is a distributed log built for Facebook sized problems.
Most of us don't have Facebook sized problems.
I don't see anything about trust requirements or verification. Does LogDevice assume that all devices in my cluster are trusted?