Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HyperDex: A searchable distributed key-value store

hyperdex.org

Post reply on HN
Print view View original

21–30 of 44 posts

Re: HyperDex: A searchable distributed key-value store

#21

Post by latch »

FWIW, I find your website suffers the same problem as Riak's, which is to say, it is entirely too focused on the product rather than how _I_ use the product. It's like it's written for people who'd be interested in writing their own data store, rather than someone, like most of us, who just want to CRUD in some shape or another.

I feel like I'm reading a white paper.

Redis' website is the exact opposite. Similarly, MongoDB's website, while a far cry from Redis', is still much, much better than Riaks.

Re: HyperDex: A searchable distributed key-value store

#22

Post by egs »

gizzlon wrote: post #18
rescrv wrote: post #13

Earlier quoted context omitted.

We have a full 14 pages ( http://hyperdex.org/papers/hyperdex.pdf ) describing the tradeoffs that went into the design and where the system's strengths and weaknesses lie. Our statements that it is faster derive directly from our observations and evaluation in the paper.

That may be so, but people might dismiss everything as bs and before getting to the paper.

A real hacker would never be dismissive of code and claims backed by an open git repo and documentation.

That said, HyperDex outperforms and provides stronger guarantees than previous key-value stores due to two architectural differences.

First is a new way to distribute data called "hyperspace hashing," whose description requires a picture, and can be found here: http://hyperdex.org/about/. This is quite different from the simple bucketing performed by memcached, Dynamo, Cassandra, Riak, MongoDB and others. See the latest slide set for an illustration of differences.

Second, HyperDex maintains replicas through a technique known as "value-dependent chaining." This enables the system to keep the replicas in sync without having to pay high overheads for coordination.

Re: HyperDex: A searchable distributed key-value store

#23

Post by egs »

latch wrote: post #21

FWIW, I find your website suffers the same problem as Riak's, which is to say, it is entirely too focused on the product rather than how _I_ use the product. It's like it's written for people who'd be interested in writing their own data store, rather than someone, like most of us, who just want to CRUD in some shape or another. I feel like I'm reading a white paper. Redis' website is the exact opposite. Similarly, M…

Thanks, this is a valid point.

Do take a look at our basic tutorial: http://hyperdex.org/doc/tutorial/

Then the more advanced one showing asynchronous operations, atomic operations, and fault tolerance: http://hyperdex.org/doc/tutorial-advanced/

And finally the rich DB interface that supports lists, sets and dictionaries, each with supporting asynchronous and atomic operations: http://hyperdex.org/doc/tutorial-datastructure/

to see which, if any, of the features and API might be a good fit for your specific needs.

Re: HyperDex: A searchable distributed key-value store

#24

Post by rdtsc »

rescrv wrote: post #12
rdtsc wrote: post #7

Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides http://hyperdex.org/slides/2012-04-13-HyperDex.pdf : --- * Consistent: linearizable. GET returns latest PUT, always * Available: in the presence of  * Partition-Tolerant: for partitions with  --- So it seems they have strong C and a tunable trade-off between A and P ? Then they take a jab at the popular interpretation of the…

CAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem ( http://hyperdex.org/extras/ ). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If…

Thanks for explaining. Your project looks very interesting. I will be definitely following it.

Can you explain some more about how node failure are handled? And whether there is a way to tune that. I.e more about the phrase "less than f nodes fail at any give point in time".

Is that tunable and how dynamic is that. (Set once or dynamic setting? Also set per running cluster, per name space, per key-value pair?)

Also, how is this f number related to the total number of machines in the cluster and how is it related to the number of data replicas?

Re: HyperDex: A searchable distributed key-value store

#25

Post by rescrv »

rdtsc wrote: post #24
rescrv wrote: post #12

Earlier quoted context omitted.

CAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem ( http://hyperdex.org/extras/ ). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If…

Thanks for explaining. Your project looks very interesting. I will be definitely following it. Can you explain some more about how node failure are handled? And whether there is a way to tune that. I.e more about the phrase "less than f nodes fail at any give point in time". Is that tunable and how dynamic is that. (Set once or dynamic setting? Also set per running cluster, per name space, per key-value pair?) Also,…

f is the number of failures that the system can tolerate. Typical systems can tolerate f failures with either (f+1) or (2f + 1) replicas.

HyperDex can tolerate f failures with f+1 replicas. This failure threshold is per region of the hyperspace. Within each region, servers are arranged in a chain. As servers fail they are removed from the chain. As new servers come into the cluster they are appended to the end of one or more chains. The "less than f nodes fail at any given point in time" translates to "at least one node in the chain is alive". Notice that chains are of length f+1 while only f nodes can fail, guaranteeing that one node is alive.

This will be tunable parameter. Right now, it is set when a new space is created, but in an upcoming release this will be totally dynamic and changeable on a per-region (of the hyperspace) basis.

I'd recommend f values of at most three. If failures are totally independent, this will be sufficient to last for millions of years. Thus, it is independent of the number of machines (once you have a sufficient number of machines).

Re: HyperDex: A searchable distributed key-value store

#27

Post by Diederich »

rescrv wrote: post #12
rdtsc wrote: post #7

Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides http://hyperdex.org/slides/2012-04-13-HyperDex.pdf : --- * Consistent: linearizable. GET returns latest PUT, always * Available: in the presence of  * Partition-Tolerant: for partitions with  --- So it seems they have strong C and a tunable trade-off between A and P ? Then they take a jab at the popular interpretation of the…

CAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem ( http://hyperdex.org/extras/ ). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If…

I was guilty, for a time, of flinging 'CAP' around. Matters are, I believe, both simpler and far more complex.

Here's the simplest way I can imagine a scenario. This is very similar to what my company deals with all the time.

Given two, network distant data centers. Requests can land on either datacenter. A request can be a PUT or a GET.

We deal with phone calls, so we have to provide an exceptionally reliable and consistent service.

The problem: a PUT lands in datacenter A. All subsequent GETs have to be consistent with this PUT. Even if they land in datacenter B.

The rest of the problem: datacenters have to be able to function independently. This means that we can't block a PUT while it sync's to the other datacenter. The other datacenter might be down, or there might be some network delay, or whatever.

In my mind, it's simply impossible to have complete consistency and complete reliability when the redundant pieces are WAN connected.(1) One can only approach and approximate this, with more and more hardware, circuits and development complexity.

Is my assertion incorrect? I'm unclear if HyperDex helps me with this problem.

By the way, I'm pretty excited about this product overall. If I wasn't in catch-up mode at work, I'd be hammering out a proof of concept project on it right away. It seems pretty compelling.

And thanks for this offering!

1) I believe a 'thick client' can, almost, solve this problem. Consider some Javascript web app. It can be developed to transparently handle various network partition situations. More or less. But many of our requests allow absolutely no code on the client side. They are standard REST calls.

Re: HyperDex: A searchable distributed key-value store

#28

Post by DennisP »

rescrv wrote: post #20
DennisP wrote: post #17

Earlier quoted context omitted.

I'd be very interested in your threaded-discussion sample. Couldn't find it on your site, do you have it online somewhere?

We have a demo of an earlier version deployed on http://gibbr.org/ . We currently are not releasing the source, but we may make an example application using the same design we use in the real app.

Cool. An article on how it's designed might be just as good.

Re: HyperDex: A searchable distributed key-value store

#29

Post by DennisP »

rescrv wrote: post #25
rdtsc wrote: post #24

Earlier quoted context omitted.

Thanks for explaining. Your project looks very interesting. I will be definitely following it. Can you explain some more about how node failure are handled? And whether there is a way to tune that. I.e more about the phrase "less than f nodes fail at any give point in time". Is that tunable and how dynamic is that. (Set once or dynamic setting? Also set per running cluster, per name space, per key-value pair?) Also,…

f is the number of failures that the system can tolerate. Typical systems can tolerate f failures with either (f+1) or (2f + 1) replicas. HyperDex can tolerate f failures with f+1 replicas. This failure threshold is per region of the hyperspace. Within each region, servers are arranged in a chain. As servers fail they are removed from the chain. As new servers come into the cluster they are appended to the end of one…

"If failures are totally independent"...that's sometimes a big if, especially on cloud providers.

I didn't see anything in the documentation about backups...what are the options?

Re: HyperDex: A searchable distributed key-value store

#30

Post by Diederich »

Diederich wrote: post #27
rescrv wrote: post #12

Earlier quoted context omitted.

CAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem ( http://hyperdex.org/extras/ ). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If…

I was guilty, for a time, of flinging 'CAP' around. Matters are, I believe, both simpler and far more complex. Here's the simplest way I can imagine a scenario. This is very similar to what my company deals with all the time. Given two, network distant data centers. Requests can land on either datacenter. A request can be a PUT or a GET. We deal with phone calls, so we have to provide an exceptionally reliable and co…

Ok, I found these slides and I believe we are on the same page, so to speak:

http://hyperdex.org/slides/2012-04-13-HyperDex.pdf

Quote:

What CAP really says: - if you cannot constrain your faults in any way, - and your requests can be directed at any server, - and you insist on serving every request, - then you cannot possibly be consistent.

No shortage of systems that preemptively give up on C, A, and P, especially C

End quote.

I think that's exactly correct.

Which leaves me with a nearly impossible problem. But it sure is fun to work on!

Post reply on HN