How does it compare to elasticsearch?
HyperDex: A Searchable Distributed Key-Value Store
31–40 of 89 posts
Re: HyperDex: A Searchable Distributed Key-Value Store
#32Since they compare it with redis, I wonder if this can handle data bigger than memory. The other properties seem nice :)
We compared it to Redis as it is one of the many key-value/document-database systems that provide the high throughput and low latency properties that make it comparable to HyperDex. We are expanding the systems we benchmark against, aiming to compare against as many systems as possible. HyperDex is designed for bigger-than-memory data.
The comparison with Redis is a bit strange at a first glance, what I mean is, if in Redis I replace the actual fetching of data with "return 'foo'" at every query it still returns 150k/requests per second for every operation. The same will do memcached per single core. So either in your code there is some good trick to improve speed in the networking layer, or you are comparing multiple cores/nodes to single core/node.
About benchmark "E", this is a primitive that Redis lacks, so it is in my opinion a bit misleading.
Another question is, in the benchmarks with Redis, does the data set fits inside Ram also in HyperDex DB (or OS cache)? In such a case you are comparing two in memory databases in practical terms, but at the same time you say that HyperDex is suitable for datasets bigger than RAM (in a portion of text near the Redis benchmark), but the numbers that the users will experiment in actual production setups will be different if the data set does not fit in RAM.
Finally, in the benchmark it is not clear that Redis can perform a lot of work in a single operation, for example a variadic LPUSH can insert 10 elements inside a list in the same time an LPUSH does this with 1 elmenet, more or less. If you multiply this for N cores in a stand alone Linux box you get 1 billion operations per second or alike, but I would be not comfortable with writing this in the Redis home page. I mean, if you don't specify very well the methodology this unfortunately is just marketing material without real interest for the field.
Re: HyperDex: A Searchable Distributed Key-Value Store
#33Did you compare the performance with (local) BerkeleyDB?
Also, is there a locking mechanism?
I'm the author of Torrus (torrus.org), and BerkeleyDB stability and non-network nature are quite painful. But I'm relying on its speed, concurrent locking, and some ways to acquire an exclusive lock on a table. It would be interesting to offer an alternative backend for torrus.
Re: HyperDex: A Searchable Distributed Key-Value Store
#34Earlier quoted context omitted.
The only scenario . In essentially every case of a node failing in an active database, clients will be communicating with it. When talking about durability, you have to assume things are failing in operation.
HyperDex utilizes value-dependent chains for replication. Updates move forward in the chains, while acknowledgements flow in reverse. To issue a PUT or a GET, the client contacts the head of the chain responsible for the object it is modifying/accessing. If other nodes in the chain fail, the chain will transparently recover. If the point leader fails (the head of the chain), then the client does not know if the opera…
What happens if a crash happens part way through the acknowledgement chain?
For example, in your insert, if the node containing "x" crashes before it receives the ACK from the node containing "y" - do the dangling "y" and "z" insertions ever need to be cleaned up?
Re: HyperDex: A Searchable Distributed Key-Value Store
#35Earlier quoted context omitted.
It's a bit of a middle ground. Yes, the replication is synchronous, which impacts availability. However, the master can remove a failed replica from the chain fairly quickly. In principle, with proper tuning, a node failure would merely cause a brief hiccup. This would feel more like a period of increased latency than a full-blown outage. So there really needn't be much sacrifice of availability. However, there's als…
This sounds like a CP system to me. There's nothing wrong with that btw, I don't know why people are so reluctant to admit this. AP systems have some useful properties, but they're also (typically) more difficult to reason about. The "hiccups" you describe are periods of unavailability. The increased latency is caused by an element of the system waiting for the data to become available again, a totally valid strategy…
To be precise, I would argue this is actually a CA system.
"C" because there are consistency guarantees, which are upheld even in the face of failures / partition.
"A" because the system will continue making progress even after a node failure. What I called a "hiccup" can be made arbitrarily short, in principle at least. The system can work around failed nodes, it does not need to wait for them to be repaired.
Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress.
Re: HyperDex: A Searchable Distributed Key-Value Store
#36the FAQ mentions that you target x86_64 architecture.
Does the server work on i386?
also, I hope it's doing proper memory alignment and endiannes independence? (because Mongo sucks on that)
Re: HyperDex: A Searchable Distributed Key-Value Store
#37Interesting.. keep it coming. We need a unified nosql language.. Basically what SQL is.
there is sparql
Re: HyperDex: A Searchable Distributed Key-Value Store
#38Earlier quoted context omitted.
This sounds like a CP system to me. There's nothing wrong with that btw, I don't know why people are so reluctant to admit this. AP systems have some useful properties, but they're also (typically) more difficult to reason about. The "hiccups" you describe are periods of unavailability. The increased latency is caused by an element of the system waiting for the data to become available again, a totally valid strategy…
To be precise, I would argue this is actually a CA system. "C" because there are consistency guarantees, which are upheld even in the face of failures / partition. "A" because the system will continue making progress even after a node failure. What I called a "hiccup" can be made arbitrarily short, in principle at least. The system can work around failed nodes, it does not need to wait for them to be repaired. Not "P…
> Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress.
Not be able to progress means it is not available.
Re: HyperDex: A Searchable Distributed Key-Value Store
#39Earlier quoted context omitted.
We compared it to Redis as it is one of the many key-value/document-database systems that provide the high throughput and low latency properties that make it comparable to HyperDex. We are expanding the systems we benchmark against, aiming to compare against as many systems as possible. HyperDex is designed for bigger-than-memory data.
The comparison with Redis is a bit strange at a first glance, what I mean is, if in Redis I replace the actual fetching of data with "return 'foo'" at every query it still returns 150k/requests per second for every operation. The same will do memcached per single core . So either in your code there is some good trick to improve speed in the networking layer, or you are comparing multiple cores/nodes to single core/no…
Both systems were single process per host.
I agree that Redis does not have good support for "E", so anyone looking at the results should look at A-D,F.
Both systems performed entirely out of main memory to avoid touching disk.
The Redis benchmark results only came in last night. We'll be writing up the methodology soon. We used the YCSB Redis binding from branch master. I'd be interested in your opinion on how well the bindings map to what is truly best for Redis.
Re: HyperDex: A Searchable Distributed Key-Value Store
#40How does it compare to elasticsearch?
Elasticsearch is a Fulltext search solution not a K/V Database.