Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TAPIR: A new high-performance, transactional key-value store

github.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: TAPIR: A new high-performance, transactional key-value store

#21

Post by nickpsecurity »

msackman wrote: post #18
nickpsecurity wrote: post #4

"enabling TAPIR to provide the same transaction model and consistency guarantees as existing systems, like Spanner, with better latency and throughput." Oh, hell yeah! Now that's great stuff. Can't wait to see the next step done by this or another team: building an alternative to the F1 RDBMS that Google built on Spanner. Would give CochroachDB some competition.

I'm the author of GoshawkDB. I've just been watching the talk on this at https://www.youtube.com/watch?v=yE3eMxYJDiE . GoshawkDB has a very similar design wrt the messaging and replication design. In fact, in some places, it appears that GoshawkDB's design is a little simpler. There are obviously many differences too: for example GoshawkDB runs transactions on the client, GoshawkDB uses Paxos Synod instead of 2PC, an…

Very interesting. I went to search for your stuff only to find I had this bookmark:

https://news.ycombinator.com/item?id=10778946

So, I knew it was interesting and possibly great work but didn't have any time to look at it. I'll move it up the backlog a bit. Maybe at least give the papers and stuff a good skimming tonight. :)

Note: Parallel, ongoing, similar work going unnoticed is a huge problem in both IT and INFOSEC. I have a meme here about how we constantly reinvent the same solutions, make the same mistakes, or move at a slower pace due to lack of connections with similar work. I keep proposing something that's like a free and open combo of ACM or IEEE with members who are academics, FOSS coders, pro's... people that contribute at least in writing. Stash papers, code, and forums there. So, odds of serendipity increase. Thoughts?

Re: TAPIR: A new high-performance, transactional key-value store

#22

Post by msackman »

nickpsecurity wrote: post #21
msackman wrote: post #18

Earlier quoted context omitted.

I'm the author of GoshawkDB. I've just been watching the talk on this at https://www.youtube.com/watch?v=yE3eMxYJDiE . GoshawkDB has a very similar design wrt the messaging and replication design. In fact, in some places, it appears that GoshawkDB's design is a little simpler. There are obviously many differences too: for example GoshawkDB runs transactions on the client, GoshawkDB uses Paxos Synod instead of 2PC, an…

Very interesting. I went to search for your stuff only to find I had this bookmark: https://news.ycombinator.com/item?id=10778946 So, I knew it was interesting and possibly great work but didn't have any time to look at it. I'll move it up the backlog a bit. Maybe at least give the papers and stuff a good skimming tonight. :) Note: Parallel, ongoing, similar work going unnoticed is a huge problem in both IT and INFOS…

Heh, definitely! I could ramble for hours about how there are so many disincentives to share anything until launch - the whole industry really isn't set up to actually make progress. However, multiple independent co-invention of stuff is still valuable from a validation point of view. The worst bits I see are when something from 20 years ago gets reinvented. I have emailed Irene just to see if there's anything or anyway to collaborate.

Re: TAPIR: A new high-performance, transactional key-value store

#25

Post by iyzhang »

imaginenore wrote: post #19

You claim it's high-performance, and you provide no benchmarks. How does it compare with Redis, Aerospike, Tarantool, Couchbase/MemBase, Memacached, VoltDB, LevelDB, Kyoto Cabinet, Riak, Cassandra, RocksDB, LMDB, Neo4j, HBase, ArangoDB, Voldemont, FoundationDB?

Figure 14 in the paper gives comparisons with MongoDB, Redis and Cassandra using YCSB. The git repo has the YCSB bindings for TAPIR, so you can run TAPIR against any of the other systems that also have YCSB bindings.

Re: TAPIR: A new high-performance, transactional key-value store

#26

Post by iyzhang »

TallGuyShort wrote: post #17
ComNik wrote: post #13

Earlier quoted context omitted.

It means that operations don't get executed in the same order on every replica. If the systems state depends on the ordering of operations, then each replica might be in a different state for the same set of operations. "Consistent replication" would be using a protocol like Paxos to have the replicas decide on a single order of operations.

Huh... isn't that required for being "transactional"? I should probably read the paper...

No, that was our key insight. Basically, you can think of it as, using something like Paxos requires all replicas to run the same code. So every replica checks for transaction conflicts. However, you actually only need one replica to detect a conflict, so there is no reason for every replica to run the same checks -- this is wasted work. Instead, we just make sure that each conflict is checked by at least one replica.

The other interesting conclusion is that there are other workloads like this. For example, it is possible to build a reliable and better performing lock server in this way as well (and there is an implementation in the github repo). So, you'd get something similar to Chubby, but where the latency to the server is only a single round-trip in the common case.

Re: TAPIR: A new high-performance, transactional key-value store

#27

Post by imaginenore »

iyzhang wrote: post #25
imaginenore wrote: post #19

You claim it's high-performance, and you provide no benchmarks. How does it compare with Redis, Aerospike, Tarantool, Couchbase/MemBase, Memacached, VoltDB, LevelDB, Kyoto Cabinet, Riak, Cassandra, RocksDB, LMDB, Neo4j, HBase, ArangoDB, Voldemont, FoundationDB?

Figure 14 in the paper gives comparisons with MongoDB, Redis and Cassandra using YCSB. The git repo has the YCSB bindings for TAPIR, so you can run TAPIR against any of the other systems that also have YCSB bindings.

Thanks. If it really outperforms Cassandra, it's pretty impressive.

Re: TAPIR: A new high-performance, transactional key-value store

#28

Post by nickpsecurity »

msackman wrote: post #22
nickpsecurity wrote: post #21

Earlier quoted context omitted.

Very interesting. I went to search for your stuff only to find I had this bookmark: https://news.ycombinator.com/item?id=10778946 So, I knew it was interesting and possibly great work but didn't have any time to look at it. I'll move it up the backlog a bit. Maybe at least give the papers and stuff a good skimming tonight. :) Note: Parallel, ongoing, similar work going unnoticed is a huge problem in both IT and INFOS…

Heh, definitely! I could ramble for hours about how there are so many disincentives to share anything until launch - the whole industry really isn't set up to actually make progress. However, multiple independent co-invention of stuff is still valuable from a validation point of view. The worst bits I see are when something from 20 years ago gets reinvented. I have emailed Irene just to see if there's anything or any…

Great! It's awesome to see you taking the initiative! Might be an exception in the making.

Im about to leave for work but quick question. What I want to see is an open equivalent of Google's F1 RDBMS: the best one. Does yours already provide its attributes, is it more like Spanner jnstead, or what? Aside from CochroachDB, where is OSS on a F1 competitor?

Re: TAPIR: A new high-performance, transactional key-value store

#29

Post by brazzledazzle »

imaginenore wrote: post #27
iyzhang wrote: post #25

Earlier quoted context omitted.

Figure 14 in the paper gives comparisons with MongoDB, Redis and Cassandra using YCSB. The git repo has the YCSB bindings for TAPIR, so you can run TAPIR against any of the other systems that also have YCSB bindings.

Thanks. If it really outperforms Cassandra, it's pretty impressive.

I don't think there's any claims that it did. The paper and the talk both made it clear that Cassandra outperformed it. I think MongoDB is the only one it beat. But I think the point is that it's able to perform well while being consistent, not that it's faster than "weakly consistent" storage systems like Cassandra.

Re: TAPIR: A new high-performance, transactional key-value store

#30

Post by msackman »

nickpsecurity wrote: post #28
msackman wrote: post #22

Earlier quoted context omitted.

Heh, definitely! I could ramble for hours about how there are so many disincentives to share anything until launch - the whole industry really isn't set up to actually make progress. However, multiple independent co-invention of stuff is still valuable from a validation point of view. The worst bits I see are when something from 20 years ago gets reinvented. I have emailed Irene just to see if there's anything or any…

Great! It's awesome to see you taking the initiative! Might be an exception in the making. Im about to leave for work but quick question. What I want to see is an open equivalent of Google's F1 RDBMS: the best one. Does yours already provide its attributes, is it more like Spanner jnstead, or what? Aside from CochroachDB, where is OSS on a F1 competitor?

So GoshawkDB doesn't have an SQL engine currently, so in that way it's probably not comparable with F1. GoshawkDB stores and accesses an object graph. Hopefully the howtos on the website will help people get into the mindset.

I'm not sure if it's worth trying to compare anything to Spanner or F1 because unless I'm mistaken, no one outside of Google can use F1 or Spanner - they're not released. So who knows what the performance of these things actually is? There's no way to verify the claims made in any Google paper.

Post reply on HN