Would be more believable if you gave some examples of the tradeoffs that went into the design, example use-cases, strengths and weaknesses, etc..
As it stands, it just reads: "It's faster!! It's better!! It's webscale!"
HyperDex: A searchable distributed key-value store
11–20 of 44 posts
Re: HyperDex: A searchable distributed key-value store
#12Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides http://hyperdex.org/slides/2012-04-13-HyperDex.pdf : --- * Consistent: linearizable. GET returns latest PUT, always * Available: in the presence of * Partition-Tolerant: for partitions with --- So it seems they have strong C and a tunable trade-off between A and P ? Then they take a jab at the popular interpretation of the…
CAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem (http://hyperdex.org/extras/). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If you separate all clients from servers, the clients cannot contact the servers."
If, instead, you make assumptions about what kind of failures are actually possible, then you can do more. HyperDex assumes that if you have less than f nodes fail at any give point in time. A failure could be the process crashing, the machine being turned off, or a partition occurring within the datacenter. So long as this assumption is true, then HyperDex is able to survive the partition, make progress (remain available), and provide strong consistency while doing so.
By making additional assumptions, we are able to make additional guarantees. This is not unlike what other engineers do (warning: massive simplification). A engineer building a bridge can assume the bridge can hold only so many cars and that each car has an upper bound on its weight. The product of these gives the entire weight the bridge must withstand. If the engineer builds the bridge to withstand the expected weight, plus some extra, then the bridge will work in all anticipated scenarios. It will not, of course, withstand bumper-to-bumper traffic consisting of 18-wheelers filled with lead bricks. But again, it doesn't have to.
Re: HyperDex: A searchable distributed key-value store
#13Would be more believable if you gave some examples of the tradeoffs that went into the design, example use-cases, strengths and weaknesses, etc.. As it stands, it just reads: "It's faster!! It's better!! It's webscale!"
We have a full 14 pages (http://hyperdex.org/papers/hyperdex.pdf) describing the tradeoffs that went into the design and where the system's strengths and weaknesses lie.
Our statements that it is faster derive directly from our observations and evaluation in the paper.
Re: HyperDex: A searchable distributed key-value store
#14Do you have examples of use in production? Could not find on your site. Adding such info would be beneficial.
We have worked with LinkedIn to use HyperDex for some of their analytics work.
Re: HyperDex: A searchable distributed key-value store
#15This looks very interesting; however, I do wish they would stop doling out the koolaide on (almost) every page. Instead of telling me why they are "this much faster than xyz" all I see are some graphs and numbers, which isn't particularly helpful because the benchmarks could be flawed!
all I see are some graphs and numbers What the hell? You are free to disagree[1] with their results or point out flaws in their methodology. But how on earth can you complain about them providing real numbers and the code used to generate them? This is rare enough, most other databases only provide the kool-aid, without any numbers whatsoever (cf. MemSQL). [1] https://groups.google.com/d/msg/redis-db/N4oy3lCngsU/NQUw…
Thanks for the note. We were flabbergasted to read the parent comment complaining about "graphs and numbers."
Note that the issues raised by the Redis developer boil down to the following:
* The benchmark measures a primitive that Redis does not provide, so Redis looks slow: This may be true. The strength of a system lies in how well its primitives handle unanticipated uses. Undoubtedly, a system with a "do_benchmark()" interface would achieve the best results, but this is not a good way to build systems. For the record, HyperDex's interface is at the same level of abstraction as Redis's in this case.
* The benchmark compares "single-core Redis" to "multi-core HyperDex." It is true that HyperDex was designed from the ground up for a networked, multi-core system. Redis is not sharded and seems to work best when co-located on the same host as its clients. If your data fits on one host and clients are on the same machine, you should probably use Redis and not HyperDex. As for the complaint, we would have used something other than "single-core Redis" if such a thing existed. Our emails to Salvatore asking for an alternative binary went unanswered -- he chose to respond with blog entries instead of code.
* The benchmark is not real: The benchmark in question is the Yahoo Cloud Serving Benchmark. It's not something we made up. One can only imagine the kind of criticism we would get if we had actually constructed our own benchmarks. YCSB is an industry-standard benchmark commonly used to evaluate the performance of key-value stores.
These kinds of issues are really easy to resolve, without having to recourse to noise on blogs and HN: We urge everyone to test their own apps against the git repo. We worked hard to make HyperDex the best key-value store out there with the strongest properties, and we hope you find it useful.
Re: HyperDex: A searchable distributed key-value store
#16Earlier quoted context omitted.
all I see are some graphs and numbers What the hell? You are free to disagree[1] with their results or point out flaws in their methodology. But how on earth can you complain about them providing real numbers and the code used to generate them? This is rare enough, most other databases only provide the kool-aid, without any numbers whatsoever (cf. MemSQL). [1] https://groups.google.com/d/msg/redis-db/N4oy3lCngsU/NQUw…
Thanks for the note. We were flabbergasted to read the parent comment complaining about "graphs and numbers." Note that the issues raised by the Redis developer boil down to the following: * The benchmark measures a primitive that Redis does not provide, so Redis looks slow: This may be true. The strength of a system lies in how well its primitives handle unanticipated uses. Undoubtedly, a system with a "do_benchmark…
Just a further note: Although HyperDex is multithreaded by default,
when benchmarking it against Redis, we disabled all but one thread
from serving network requests.
Edit: reword for clarity.
Re: HyperDex: A searchable distributed key-value store
#17This looks interesting, but can someone explain the benefits over memcache/Redis? Is "hyperspace hashing" storing multiple copies, so it's like storing records on multiple shards of a database? And "enables lookups of non-primary data attributes": how useful is this, actually? Is this a big step forward for NoSQL? The site doesn't seem to be giving me a real-world case where HyperDex solves existing problems better t…
Memcache is a caching solution that stores binary blobs. HyperDex stores data persistently and offers a wide variety of types such as lists, sets, and maps. Redis offers many datastructures as well, but it has a limited architecture. If you want to run multiple Redis instances, you must run them in a master-slave configuration with no guarantees when the master fails. HyperDex can withstand such faults while guarante…
I'd be very interested in your threaded-discussion sample. Couldn't find it on your site, do you have it online somewhere?
Re: HyperDex: A searchable distributed key-value store
#18Would be more believable if you gave some examples of the tradeoffs that went into the design, example use-cases, strengths and weaknesses, etc.. As it stands, it just reads: "It's faster!! It's better!! It's webscale!"
We have a full 14 pages ( http://hyperdex.org/papers/hyperdex.pdf ) describing the tradeoffs that went into the design and where the system's strengths and weaknesses lie. Our statements that it is faster derive directly from our observations and evaluation in the paper.
That may be so, but people might dismiss everything as bs and before getting to the paper.
Re: HyperDex: A searchable distributed key-value store
#19Earlier quoted context omitted.
Memcache is a caching solution that stores binary blobs. HyperDex stores data persistently and offers a wide variety of types such as lists, sets, and maps. Redis offers many datastructures as well, but it has a limited architecture. If you want to run multiple Redis instances, you must run them in a master-slave configuration with no guarantees when the master fails. HyperDex can withstand such faults while guarante…
I'd be very interested in your threaded-discussion sample. Couldn't find it on your site, do you have it online somewhere?
Here it is http://gibbr.org/. An undergrad implemented this on HyperDex in about half a semester.
Re: HyperDex: A searchable distributed key-value store
#20Earlier quoted context omitted.
Memcache is a caching solution that stores binary blobs. HyperDex stores data persistently and offers a wide variety of types such as lists, sets, and maps. Redis offers many datastructures as well, but it has a limited architecture. If you want to run multiple Redis instances, you must run them in a master-slave configuration with no guarantees when the master fails. HyperDex can withstand such faults while guarante…
I'd be very interested in your threaded-discussion sample. Couldn't find it on your site, do you have it online somewhere?
We have a demo of an earlier version deployed on http://gibbr.org/. We currently are not releasing the source, but we may make an example application using the same design we use in the real app.