Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HyperDex: A searchable distributed key-value store

hyperdex.org

Post reply on HN
Print view View original

1–10 of 44 posts

Re: HyperDex: A searchable distributed key-value store

#3

Post by Ixiaus »

This looks very interesting; however, I do wish they would stop doling out the koolaide on (almost) every page. Instead of telling me why they are "this much faster than xyz" all I see are some graphs and numbers, which isn't particularly helpful because the benchmarks could be flawed!

Re: HyperDex: A searchable distributed key-value store

#5

Post by crazygringo »

This looks interesting, but can someone explain the benefits over memcache/Redis?

Is "hyperspace hashing" storing multiple copies, so it's like storing records on multiple shards of a database?

And "enables lookups of non-primary data attributes": how useful is this, actually? Is this a big step forward for NoSQL?

The site doesn't seem to be giving me a real-world case where HyperDex solves existing problems better than other things... maybe I just can't find it?

Re: HyperDex: A searchable distributed key-value store

#6

Post by rescrv »

Ixiaus wrote: post #3

This looks very interesting; however, I do wish they would stop doling out the koolaide on (almost) every page. Instead of telling me why they are "this much faster than xyz" all I see are some graphs and numbers, which isn't particularly helpful because the benchmarks could be flawed!

The code is available on GitHub. If the benchmark is flawed, help us fix it.

Here are some ways it could be flawed:

* MongoDB and Cassandra were set to default consistency levels. They are not configured to wait for the majority of replicas to respond. HyperDex waits until data is committed at all replicas. This clearly biases the benchmark away from HyperDex.

* MongoDB and Cassandra operate on the key for the search benchmark, while HyperDex operates (solely) on secondary attributes. Another bias against HyperDex.

Most of our gain in the benchmarks comes from high GET throughput and high SEARCH throughput, although PUTs are competitive as well.

We are faster than MongoDB because they do idiotic stuff they don't have to. When I was investigating why HyperDex is faster, I looked solely at client libraries (since MongoDB's default config just writes to socket buffers, or buffers it in userspace). HyperDex has one function to create the request packet, one function that enqueues it with a constant number of operations, and one function to flush the queue. Once a request is created it is not copied until the kernel moves it to the a socket buffer. MongoDB, on the other hand, bounces through half a dozen different layers, some of which perform memmove to compact the data, keeping it contiguous in memory. While I've not examined the whole MongoDB code base, I suspect that it's more of the same. I can tell you first hand that the same diligence paid to making the HyperDex client efficient was paid at all layers of the HyperDex stack.

Edit: Trying to get bullets to work

Re: HyperDex: A searchable distributed key-value store

#7

Post by rdtsc »

Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides

http://hyperdex.org/slides/2012-04-13-HyperDex.pdf :

---

* Consistent: linearizable. GET returns latest PUT, always

* Available: in the presence of  * Partition-Tolerant: for partitions with  ---

So it seems they have strong C and a tunable trade-off between A and P ?

Then they take a jab at the popular interpretation of the CAP theorem and claim they have a work-around:

----

   Working around the [CAP] theorem:
     * Constrain the failure size
     * Redirect clients to majority partition
     * Profit: Retain all of C, A, P
   Realistic for a modern data center

   CAP misses the point The real tradeoff is between C, A, and Performance.
---

Not sure what they meant here. Anyone understood this better?

Re: HyperDex: A searchable distributed key-value store

#8

Post by moe »

Ixiaus wrote: post #3

This looks very interesting; however, I do wish they would stop doling out the koolaide on (almost) every page. Instead of telling me why they are "this much faster than xyz" all I see are some graphs and numbers, which isn't particularly helpful because the benchmarks could be flawed!

all I see are some graphs and numbers

What the hell?

You are free to disagree[1] with their results or point out flaws in their methodology. But how on earth can you complain about them providing real numbers and the code used to generate them?

This is rare enough, most other databases only provide the kool-aid, without any numbers whatsoever (cf. MemSQL).

[1] https://groups.google.com/d/msg/redis-db/N4oy3lCngsU/NQUwf12...

Re: HyperDex: A searchable distributed key-value store

#9

Post by rescrv »

crazygringo wrote: post #5

This looks interesting, but can someone explain the benefits over memcache/Redis? Is "hyperspace hashing" storing multiple copies, so it's like storing records on multiple shards of a database? And "enables lookups of non-primary data attributes": how useful is this, actually? Is this a big step forward for NoSQL? The site doesn't seem to be giving me a real-world case where HyperDex solves existing problems better t…

Memcache is a caching solution that stores binary blobs. HyperDex stores data persistently and offers a wide variety of types such as lists, sets, and maps.

Redis offers many datastructures as well, but it has a limited architecture. If you want to run multiple Redis instances, you must run them in a master-slave configuration with no guarantees when the master fails. HyperDex can withstand such faults while guaranteeing linearizable semantics. Further, we've got some changes in the pipeline that will enable us to support every datastructure operation Redis supports (BRPOPLPUSH, I'm looking at you) with horizontal partitioning across multiple machines.

Hyperspace hashing stores multiple copies, but this is configured by the user. In our applications we've had at most two copies.

You can find more about looking up non-primary data attributes in the tutorial (http://hyperdex.org/doc/tutorial/#creating-a-new-space). It enables you to search over attributes of an object without having to maintain secondary indices.

We've built a sample threaded-discussion application on top of HyperDex. We're also in talks with some astronomers who would like to use it to analyze data from a radio telescope. The geometric nature of hyperspace hashing makes it easy to perform queries such as, "retrieve all objects in this cone through space." For any application where you perform secondary attribute search, HyperDex is the way to go. For other applications where you want high throughput and low latency with strong consistency, HyperDex is a good bet.

Post reply on HN