Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

databeta.wordpress.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#31

Post by bpicolo »

xstartup wrote: post #11
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sor…

You can disk-back redis at various consistency levels if you so choose

https://redis.io/topics/persistence

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#32

Post by ricardobeat »

nicpottier wrote: post #21

Being faster (in throughput) than Redis doesn't seem that difficult, Redis is mostly single core. But with that restriction comes the magical constraint that you can run Lua on that single core without having to think about race conditions or concurrency. For many complicated systems this is an incredible property and can provide incredible power. Yes, you may eventually outgrow Redis, but if you do then you are ente…

The comparison with Redis is on page 10, and (obviously) takes into account the number of threads. It is a bit presumptuous to post a comment like this without minimal effort, please don’t.

Regarding embedded processing, it’s very easy to embed whatever single-threaded language you’d like into C: Lua, Javascript or something new like Gravity. This is orthogonal to the storage / network architecture.

A better argument would be on the extra operations and data structures that Redis offers, not being a simple key-value store.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#33

Post by digikata »

SanFranManDan wrote: post #20
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself. There really aren't even that many DBMS that are KV (like redis) out…

I think we're just getting to the point where it may become more common to separate the simpler problem of a single node, non-distributed 'store' with some kv interface below, and then build more complex distributed algorithms in a layer or two above. You can see some of the larger monolithic codebases that had to start by having their own code all the way up and down the stack, but now are starting to experiment with backend store interfaces so you can trade-off some of the strengths and weaknesses of various local store performance areas.

Along the same lines, a few newer codebases for distributed stores seem to be building with those delineations in mind. Another comment brought up Tidb/Tikv for example. Tikv iirc uses RocksDB as its local store.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#34

Post by antoncohen »

SanFranManDan wrote: post #20
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself. There really aren't even that many DBMS that are KV (like redis) out…

RocksDB, LMDB, and LevelDB are basically low level disk representations used by the databases that do things like provide network access, sharding, and replication. OpenLDAP (LMDB), MySQL (MyRocks), Bigtable (LevelDB-like), Riak (LevelDB), etc.

Many are or can be used as key-value stores. MySQL actually has a memcached compatible KV store, using InnoDB for storage. Postgres has HStore. A lot of the distributed databases roughly fall into the category of KV stores: HBase, Riak, Cassandra, DynamoDB, etc.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#35

Post by nilkn »

xstartup wrote: post #11
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sor…

Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#36

Post by snissn »

muxator wrote: post #26

Rant: Why so much focus on Key Value stores? That's the easy part of the problem. I would like to know more about the interesting ones: secondary indexes, range scans, performance on mixed workloads, robustness, operational complexity.

> Why so much focus on Key Value stores? That's the easy part of the problem.

There aren't enough good/fast/reliable ones

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#37

Post by [deleted] »

nicpottier wrote: post #21

Being faster (in throughput) than Redis doesn't seem that difficult, Redis is mostly single core. But with that restriction comes the magical constraint that you can run Lua on that single core without having to think about race conditions or concurrency. For many complicated systems this is an incredible property and can provide incredible power. Yes, you may eventually outgrow Redis, but if you do then you are ente…

[deleted]

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#39

Post by electrum »

xstartup wrote: post #11
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sor…

> RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it?

https://myrocks.io/

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#40

Post by majidazimi »

muxator wrote: post #26

Rant: Why so much focus on Key Value stores? That's the easy part of the problem. I would like to know more about the interesting ones: secondary indexes, range scans, performance on mixed workloads, robustness, operational complexity.

Here is my 2 cent on why this is the case:

On a general purpose hardware/network adding features as you mentioned, tends to be seriously difficult and tremendously workload dependent. At some point you need help from hardware designers which itself limits the use cases of your system.

FPGA supported databases, In-network SQL processing, Server-less caches (an SSD directly connected to network without OS), Storage systems with mind blowing low energy consumption, ... are just bunch of absolutely amazing research that are being done today. however the limited public availability is badgering.

With mass public usability in-mind, a general purpose K/V system is the highest summit you can really achieve.

Post reply on HN