Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LogDevice: a distributed data store for logs

code.facebook.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: LogDevice: a distributed data store for logs

#11

Post by shock »

amelius wrote: post #10
shock wrote: post #2

> We continue to iterate on LogDevice with the ultimate goal of contributing it to the open source community later in 2017. Frankly, if it's the same license as React (BSD + PATENTS), I'm not interested. Edit: Here come the facebook fanboys with the downvotes.

Frankly, if this comes from a company that treats its users as products, I'm not interested no matter what the license says.

These days I get the impression that there are fewer and fewer companies that don't see users as products. I hope I'm wrong.

Re: LogDevice: a distributed data store for logs

#12

Post by sctb »

shock wrote: post #2

> We continue to iterate on LogDevice with the ultimate goal of contributing it to the open source community later in 2017. Frankly, if it's the same license as React (BSD + PATENTS), I'm not interested. Edit: Here come the facebook fanboys with the downvotes.

We've had many discussions about licensing issues, but this isn't one of them. Please don't post off topic and then complain about downvotes—it breaks the guidelines.

https://news.ycombinator.com/newsguidelines.html

Re: LogDevice: a distributed data store for logs

#13

Post by shock »

sctb wrote: post #12
shock wrote: post #2

> We continue to iterate on LogDevice with the ultimate goal of contributing it to the open source community later in 2017. Frankly, if it's the same license as React (BSD + PATENTS), I'm not interested. Edit: Here come the facebook fanboys with the downvotes.

We've had many discussions about licensing issues, but this isn't one of them. Please don't post off topic and then complain about downvotes—it breaks the guidelines. https://news.ycombinator.com/newsguidelines.html

I never imagined that the project's license would be off-topic. To me a project's license is very material, a deciding factor if I'm using it or not. That's what I was trying to express.

Re: LogDevice: a distributed data store for logs

#14

Post by jasonwatkinspdx »

shock wrote: post #2

> We continue to iterate on LogDevice with the ultimate goal of contributing it to the open source community later in 2017. Frankly, if it's the same license as React (BSD + PATENTS), I'm not interested. Edit: Here come the facebook fanboys with the downvotes.

I'm not a facebook fanboy, but I did vote this down because it's a derailment based on an assumption. This isn't released yet and no one knows the terms.

Atop that, I'm just getting really tired of seeing the patent issue trotted out any and every time facebook or react are mentioned here.

Re: LogDevice: a distributed data store for logs

#16

Post by prohor »

polskibus wrote: post #3

How does it differ from Apache Kafka?

For me the difference seems to be in handling large number of distinct logs. In Kafka every log & partition is a separate file and moreover it keeps it open. So, storing multiple logs results in writing to many files so eventually random write IO; and also you may hit limits of open files. You can multiplex logical logs in each Kafka log, but then you read unnecessarily other logs.

Keeping SS tables makes it more sequential write and reasonably sequential write, as long as you have enough RAM to get multiple records of each log, so they constitute a continues blocks in flashed file.

Actually you could get very similar result using Cassandra, which also uses SS tables. The difference is that Cassandra keeps merging files, which actually makes much more IO traffic than clients. Cassandra will typically need 16x more IO for merging then actual data write rate. You can limit it a bit if you create time shard tables.

Re: LogDevice: a distributed data store for logs

#17

Post by sctb »

shock wrote: post #13
sctb wrote: post #12

Earlier quoted context omitted.

We've had many discussions about licensing issues, but this isn't one of them. Please don't post off topic and then complain about downvotes—it breaks the guidelines. https://news.ycombinator.com/newsguidelines.html

I never imagined that the project's license would be off-topic. To me a project's license is very material, a deciding factor if I'm using it or not. That's what I was trying to express.

The project under discussion doesn't have a license.

Re: LogDevice: a distributed data store for logs

#18

Post by jbit84 »

Reading the blog its not clear to me how they deal with gaps in the LSN sequence. The scalability & performance properties derive from 1) Using a separate sequencer that issues increasing sequence numbers, 2) Uncoordinated distributed writes of actual record value to storage nodes, 3) reconstitution of ordered log at consumer side.

How does a consumer that have retrieved N and N+2 know if N+1 is not yet written, or if it failed and will never be written? Perhaps they write-through the sequencer with subsequent writes waiting on acknowledgements, so 'gaps' only occur on epoch changes?

Re: LogDevice: a distributed data store for logs

#19

Post by kordless »

sctb wrote: post #12
shock wrote: post #2

> We continue to iterate on LogDevice with the ultimate goal of contributing it to the open source community later in 2017. Frankly, if it's the same license as React (BSD + PATENTS), I'm not interested. Edit: Here come the facebook fanboys with the downvotes.

We've had many discussions about licensing issues, but this isn't one of them. Please don't post off topic and then complain about downvotes—it breaks the guidelines. https://news.ycombinator.com/newsguidelines.html

> we've had many

> don't post off topic

Users of HN, please note "we" is Facebook and Facebook, or a representative of Facebook, is telling us what we can and can't discuss here.

This is why I quit posting here and use /r/hackernews on Reddit for my links. However, I occasionally make an appearance when I see patterns emerge, such as control where we didn't assume control existed.

Post reply on HN