Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook open-sources LogDevice, a distributed storage for sequential data

logdevice.io

Post reply on HN
Print view View original

101–110 of 123 posts

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#101

Post by otterley »

beepbeepbeep1 wrote: post #93
otterley wrote: post #69

Earlier quoted context omitted.

> Is it because Kafka doesn't scale to millions of logs on a single cluster I doubt that's it, since Kafka can certainly do that.

It does not, I've lost alot of time profiling Kafka perf issues against clusters on the exact same hardware with exact same traffic but with a 3000% throughput difference. The root cause was one cluster had a lot of empty test topics Try benchmarking Kafka from 0 partitions to a few thousand partitions in 100 partition increments. The benchmark only needs to write to a single topic, using their provided producer perf…

I didn't think we were speaking of millions of topics or partitions here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#103

Post by noahdesu »

senderista wrote: post #102

Sounds like it might have been influenced by the MSR CORFU project (separate sequencer, write striping). Can anyone confirm?

It's hard to deny that there is at least some influence there. Like LogDevice, the zlog project [0] is influence by CORFU (separate sequencer, write striping), but both use different storage interfaces / strategies.

[0]: https://github.com/cruzdb/zlog

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#104

Post by sh00s »

otterley wrote: post #100
manigandham wrote: post #90

Earlier quoted context omitted.

Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.

I didn't think we were speaking of millions of topics here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

A log in LogDevice is roughly equivalent to a Kafka partition.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#105

Post by manigandham »

otterley wrote: post #100
manigandham wrote: post #90

Earlier quoted context omitted.

Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.

I didn't think we were speaking of millions of topics here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

Do you mean log segments then? In that case I don't see what's special about it because that's just rolling files and all of these systems can handle millions that way.

As far as millions of topics, if you have to do it at a logical layer yourself, then you might as well use a system that supports it natively.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#106

Post by rakoo »

otterley wrote: post #100
manigandham wrote: post #90

Earlier quoted context omitted.

Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.

I didn't think we were speaking of millions of topics here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

The logs in LogDevice also have an independent lifecycle, which your solution doesn't allow.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#107

Post by geeters1 »

atombender wrote: post #79
simonrobb wrote: post #74

Earlier quoted context omitted.

OKLog, Humio, and Splunk are all worth checking out.

OKLog has been abandoned by the author (the project is now read only on GitHub). Humio is not self-hosted or open source, so not really a fair comparison. It also seems targeted towards operational logs, i.e. system logging, traffic logging, auditing. Not things like data pipelines. Kafka and friends can be used for that kind of log, but they are more like databases; they use the term "log" in the sense of sequential…

Humio does have a self-hosted version - but is closed source. You can download a trial and it supports most logs types via support for opensource log shippers, ie logstash and beats..along with other popular formats including Kafka.

https://docs.humio.com/integrations/

The UI is what simplifies analysis and visualization with live, real-time query and db.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#108

Post by Annatar »

javiermaestro wrote: post #99
Annatar wrote: post #94

Earlier quoted context omitted.

It's for Linux only, and I run illumos-based SmartOS on my own infrastructure. That's not the point. The point is that all these generation Y kids grew up on PC buckets and still don't understand UNIX and the concepts behind it, and yet they use it to power their applications. This can only end badly unless they start making an effort to understand the concepts behind the substrate they are writing software for.

A few things. First, can't you run Linux inside a Solaris Zone? I don't know much about Solaris stuff (although I do like it very much, I grew up mostly with Linux, which you so much despise, and I'm not too familiar with other Unixes). So... I think you could probably run Logdevice if you really wanted. Then, here's my two cents. When engaging in conversation and civil dialog, please try to avoid being so dismissive…

"You come across as abrasive and entitled."

Does it appear to you that I care? I am abrasive. And I am entitled. I'm also very proud that I run an illumos based operating system instead of a GNU/Linux one. Anything else I can clarify for you?

"I think you could probably run Logdevice if you really wanted."

Nowhere did I write that I actually care about logdevice. What I care about is that these kids learn UNIX: daemons and system administration utilities go into sbin/, not bin/.

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#109

Post by ivankelly »

sh00s wrote: post #98
ivankelly wrote: post #97

Earlier quoted context omitted.

What happens with a sequencer which appears to fail but hasn't really, and then comes back up after the process to figure out inflight records has completed? If that sequencer receives a record for a log, will it be able to write it to the storage nodes? I.e. is there any fencing mechanism to tell the storage nodes that the epoch has been bumped, so don't access writes for that epoch anymore?

yes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and consistency LogDevice picks higher availability. Blocking new writes until sealing is complete, however, should be fairly easy to integrate into LogDevice as an option.

Does it block reads until sealing is complete? How many nodes in the nodeset have to respond before sealing is complete? [NodeSet] - [ReplicationFactor] + 1?

Re: Facebook open-sources LogDevice, a distributed storage for sequential data

#110

Post by sh00s »

ivankelly wrote: post #109
sh00s wrote: post #98

Earlier quoted context omitted.

yes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and consistency LogDevice picks higher availability. Blocking new writes until sealing is complete, however, should be fairly easy to integrate into LogDevice as an option.

Does it block reads until sealing is complete? How many nodes in the nodeset have to respond before sealing is complete? [NodeSet] - [ReplicationFactor] + 1?

Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority.

For a simple case where placement of data is location-agnostic, indeed the definition of f-majority is n - r + 1, where n is the nodeset size, and r is the replication factor.

However, if your replication property, is say, "place 3 copies across 3 racks", then the definition of f-majority becomes more complicated - e.g. having all nodes in the nodeset respond minus two racks will also satisfy it.

Post reply on HN