Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

databeta.wordpress.com

Post reply on HN
Print view View original

11–20 of 77 posts

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#11

Post by xstartup »

ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

Cassandra is JVM GC issues

Redis is in-memory only

Cloud Storage - Not sure, how we can use it outside of cloud vendors

RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced?

There is also badger but most of these only offer low-level operation.

Sorry, most of my developers are unable to consume them just like Redis.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#12

Post by hardwaresofton »

So reading through this announcement, it seems the key to ANNA's speed is:

local cache (in the form of the actor's mailbox) + background gossip

The usual restrictions (and increases in latency) still apply when you want to make sure that something's actually written (quoruming) after you've written it, from what I can tell.

Can someone explain to me why this is a step forward for the field -- I haven't yet read all the papers they linked to (including their ANNA papers), but this doens't seem to be one of those times where a bunch of disparate papers are combined into creating something truly groundbreaking?

I feel like I must be missing the point

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#13

Post by ricardobeat »

The link to bloom is broken: http://bloom-lang.net

This is a result of what appears to be a chain of ideas, papers and prototypes started more than eight years ago (see http://boom.cs.berkeley.edu - maybe even earlier, hard to tell since most older URLs are gone). I'm amazed that people are able to remain funded working on something with a very theoretical and long-term payoff, and incredibly thankful at the same time for the entities supporting this! Wish success to the team on releasing Bedrock.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#14

Post by kodablah »

xstartup wrote: post #11
ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sor…

Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#15

Post by zzzcpan »

dis-sys wrote: post #7

> Totally ordered request processing requires waiting for global consensus at each step, and thus fundamentally limits the throughput of each replica-set. really? such global consensus increases latency (network round trip plus fsync write), with a fully batched and pipelined concurrent design, when CPU cycles are being saturated in those benchmarks, why throughput is fundamentally limited by such increased latency?

Let's say we have a primitive system with total ordering and single static coordinator per each replica-set. And we want to update a global counter from every node. Now since counter is global it lives on a single replica-set and every node has to communicate with this one coordinator. Be it 4 nodes or 4000, still one coordinator. If we drop total ordering and use something like gossip protocol to communicate between nodes, we can merge this counter everywhere in the system as it propagates, eliminating all unnecessary communications and distributing communications in the network. So, yeah, global consensus fundamentally limits both throughput and latency.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#16

Post by ricardobeat »

hardwaresofton wrote: post #12

So reading through this announcement, it seems the key to ANNA's speed is: local cache (in the form of the actor's mailbox) + background gossip The usual restrictions (and increases in latency) still apply when you want to make sure that something's actually written (quoruming) after you've written it, from what I can tell. Can someone explain to me why this is a step forward for the field -- I haven't yet read all t…

I don't think that's correct. The real key to its performance seems to be the usage of distributed lattices as data structures, and the ability to perform compile-time checks that guarantee the data will be eventually consistent, both of which allow code to be completely lock-free. This comes from the CALM paper cited in the article - lots of reading to do!

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#18

Post by SanFranManDan »

kodablah wrote: post #14
xstartup wrote: post #11

Earlier quoted context omitted.

Cassandra is JVM GC issues Redis is in-memory only Cloud Storage - Not sure, how we can use it outside of cloud vendors RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced? There is also badger but most of these only offer low-level operation. Sor…

Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of…

From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful.

TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD)

From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#19

Post by jmull »

dis-sys wrote: post #7

> Totally ordered request processing requires waiting for global consensus at each step, and thus fundamentally limits the throughput of each replica-set. really? such global consensus increases latency (network round trip plus fsync write), with a fully batched and pipelined concurrent design, when CPU cycles are being saturated in those benchmarks, why throughput is fundamentally limited by such increased latency?

That's a really good question... I think the answer is something like this (just spitballing here, not trying to make a strong claim):

At some point in a system, you may need a response to one request in order to generate the next request. At this point latency affects throughput.

Also, apparently, their "lattice composition" technique let's them push concurrency to a lower level, which would allow them to avoid the overhead of doing concurrency at a higher level. What I mean is, if you have to process items one-at-a-time in a replica, then to process more items at a time, you need more replicas. But each replica has overhead -- the details depend on what technology we're talking about, but there's overhead to each replica... e.g., to utilize 4 cores on your server you might host four replicas on that server... but now you have to maintain four memory spaces as well, which takes cycles, even though all you wanted was to make use of all your cores to process items. And it might not be four cores, but 18 or 36 or perhaps 1000s. At some point, the cost of the overhead is greater than amount of benefit.

Re: Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store

#20

Post by SanFranManDan »

ralusek wrote: post #5

I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be. What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases. I don't need another key value store...

I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself.

There really aren't even that many DBMS that are KV (like redis) out there to handle it either. They are normally much more complicated (like adding SQL layer on top of it).

Post reply on HN