Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

91–100 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#91

Post by manigandham »

Serow225 wrote: post #70
EdwardDiego wrote: post #57

Earlier quoted context omitted.

> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference. > Do people just point their system journal at Kafka and wait for something to break? Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads. LogDevice looks to be very…

What's a good distributed log for 10-dev sized companies? :)

Apache Bookkeeper. Or install Apache Pulsar and get full messaging capabilities with tiering out to S3 for infinite retention.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#92

Post by manigandham »

tveita wrote: post #72
the_duke wrote: post #38

Earlier quoted context omitted.

Can you give an overview over the difference to eg Apache Kafka? It seems very similar.

From what I can see this doesn't have built-in consumer balancing and offset storage, like Kafka does. It also lacks more exotic Kafka features like topic compaction and exactly-once processing. In Kafka bulk reading is very cheap, the broker basically just calls sendfile() to send a file segment with compressed message chunks. On the other hand only the leader of a partition can serve requests, so you are often limi…

Kafka is also very embedded everywhere now, with a big first-mover advantage. Pulsar already does everything Kafka does but also supports custom functions, per-message acknowledgements, and native cross-region replication.

Unfortunately it's hard to change something that already works. Most users don't hit the performance limits of their tools so they'll just continue Kafka if it's already running.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#93

Post by beepbeepbeep1 »

otterley wrote: post #69
thinkersilver wrote: post #66

The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writ…

> Is it because Kafka doesn't scale to millions of logs on a single cluster I doubt that's it, since Kafka can certainly do that.

It does not, I've lost alot of time profiling Kafka perf issues against clusters on the exact same hardware with exact same traffic but with a 3000% throughput difference. The root cause was one cluster had a lot of empty test topics

Try benchmarking Kafka from 0 partitions to a few thousand partitions in 100 partition increments. The benchmark only needs to write to a single topic, using their provided producer perf tool while all other topics are inactive with zero data.

As the partitions increase there is a very noticeable drop in throughout that looks to be linear.

Kafka does not handle a large number of partitions well currently, large even being low thousands. It's easy to hit with just a few hundred topics.

Reading between the lines ehen Linkdin and Netflix advertise several clusters, i am predicting/guessing they shard the data.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#94

Post by Annatar »

AhmedSoliman wrote: post #51
Annatar wrote: post #50

"bin/logdeviced" All daemons and system administration utilities belong into sbin, because bin is for end-user applications. Historically, the "s" in sbin meant something else, but it always contained applications and scripts only root could run. When I see these examples, it's depressing to see just how much understanding of UNIX is missing.

Maybe sending a PR would help?

It's for Linux only, and I run illumos-based SmartOS on my own infrastructure.

That's not the point. The point is that all these generation Y kids grew up on PC buckets and still don't understand UNIX and the concepts behind it, and yet they use it to power their applications. This can only end badly unless they start making an effort to understand the concepts behind the substrate they are writing software for.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#96

Post by AhmedSoliman »

Annatar wrote: post #94
AhmedSoliman wrote: post #51

Earlier quoted context omitted.

Maybe sending a PR would help?

It's for Linux only, and I run illumos-based SmartOS on my own infrastructure. That's not the point. The point is that all these generation Y kids grew up on PC buckets and still don't understand UNIX and the concepts behind it, and yet they use it to power their applications. This can only end badly unless they start making an effort to understand the concepts behind the substrate they are writing software for.

I will let the generation Y kids know your opinion ;)

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#97

Post by ivankelly »

martincmartin wrote: post #24
ashu wrote: post #13

Earlier quoted context omitted.

How does LogDevice differ from Kafka?

I worked on LogDevice at FB until about 6 months ago. I'm not that familiar with Kafka, but in general LogDevice emphasizes write availability over read availability. There are many applications where data is being generated all the time, and if you don't write it, it will be lost. However, if reading is delayed, it just means readers are a little behind and will need to catch up. So, when a sequencer node dies and w…

What happens with a sequencer which appears to fail but hasn't really, and then comes back up after the process to figure out inflight records has completed? If that sequencer receives a record for a log, will it be able to write it to the storage nodes? I.e. is there any fencing mechanism to tell the storage nodes that the epoch has been bumped, so don't access writes for that epoch anymore?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#98

Post by sh00s »

ivankelly wrote: post #97
martincmartin wrote: post #24

Earlier quoted context omitted.

I worked on LogDevice at FB until about 6 months ago. I'm not that familiar with Kafka, but in general LogDevice emphasizes write availability over read availability. There are many applications where data is being generated all the time, and if you don't write it, it will be lost. However, if reading is delayed, it just means readers are a little behind and will need to catch up. So, when a sequencer node dies and w…

What happens with a sequencer which appears to fail but hasn't really, and then comes back up after the process to figure out inflight records has completed? If that sequencer receives a record for a log, will it be able to write it to the storage nodes? I.e. is there any fencing mechanism to tell the storage nodes that the epoch has been bumped, so don't access writes for that epoch anymore?

yes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and consistency LogDevice picks higher availability. Blocking new writes until sealing is complete, however, should be fairly easy to integrate into LogDevice as an option.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#99

Post by javiermaestro »

Annatar wrote: post #94
AhmedSoliman wrote: post #51

Earlier quoted context omitted.

Maybe sending a PR would help?

It's for Linux only, and I run illumos-based SmartOS on my own infrastructure. That's not the point. The point is that all these generation Y kids grew up on PC buckets and still don't understand UNIX and the concepts behind it, and yet they use it to power their applications. This can only end badly unless they start making an effort to understand the concepts behind the substrate they are writing software for.

A few things. First, can't you run Linux inside a Solaris Zone? I don't know much about Solaris stuff (although I do like it very much, I grew up mostly with Linux, which you so much despise, and I'm not too familiar with other Unixes). So... I think you could probably run Logdevice if you really wanted.

Then, here's my two cents. When engaging in conversation and civil dialog, please try to avoid being so dismissive and so proud of yourself and of how much you think you know about stuff. You come across as abrasive and entitled. It's not nice to just jump into a conversation and talk trash about the work of others just because you dislike the operating system that they use.

Finally, if you really care, work on porting it to your operating system of choice and engage in civil conversation doing pull requests, etc. Everybody will be thankful for that.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#100

Post by otterley »

manigandham wrote: post #90
otterley wrote: post #69

Earlier quoted context omitted.

> Is it because Kafka doesn't scale to millions of logs on a single cluster I doubt that's it, since Kafka can certainly do that.

Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.

I didn't think we were speaking of millions of topics here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.
Post reply on HN