Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster – Fast key-value store from Microsoft Research

github.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Faster – Fast key-value store from Microsoft Research

#71

Post by brongondwana »

skyde wrote: post #26

I had to opportunity to look at the code. I believe this should be compared against embedded hash database such as "kyoto cabinet, LevelDB, RocksDB" . They introduce a novel latch free hashtable that they say is faster than other in memory data structure. They also introduce a new disk persistence system called HybridLog that combines in-place updates (in memory) and log-structured organization (on disk). The interes…

I was very surprised to see pretty much the entire implementation inside a file called faster.h rather than in a .cc file. Maybe that's all the rage in C++ libraries, I don't work with many of them.

Re: Faster – Fast key-value store from Microsoft Research

#72

Post by imhoguy »

mooman219 wrote: post #3

"What differentiates FASTER are its cache-optimized index that achieves very high performance — up to 160 million operations per second when data fits in memory;" I really dislike when papers make performance claims like this in the introduction. That "160 million" number is so meaningless at face value because everything from the runtime environment to the hardware is going to play a huge role in ops. I rather see h…

This. Somebody already filled an issue https://github.com/Microsoft/FASTER/issues/2

Re: Faster – Fast key-value store from Microsoft Research

#73

Post by yarper »

atombender wrote: post #7

For those wondering what this is, it is not a client/server app, from what I can tell, but an embedded engine. It looks like it's intended to be a library, and it's been implemented in two languages (C# and C++). To get something like Redis or Riak you would have to build API, clustering, etc. on top of it. So it's more analogous to libraries like RocksDB, BoltDB, BDB etc. Paper: https://www.microsoft.com/en-us/resea…

Can anyone work out what platforms it works on? .net core?

Re: Faster – Fast key-value store from Microsoft Research

#74

Post by SideburnsOfDoom »

yarper wrote: post #73
atombender wrote: post #7

For those wondering what this is, it is not a client/server app, from what I can tell, but an embedded engine. It looks like it's intended to be a library, and it's been implemented in two languages (C# and C++). To get something like Redis or Riak you would have to build API, clustering, etc. on top of it. So it's more analogous to libraries like RocksDB, BoltDB, BDB etc. Paper: https://www.microsoft.com/en-us/resea…

Can anyone work out what platforms it works on? .net core?

Here is the relevant metadata: https://github.com/Microsoft/FASTER/blob/master/cs/src/core/...

Should be usable in NetCoreApp 2.0 and up, or .NET full framework 4.6 and up. In other words, yes, it will work on recent versions of NetCore and classic .Net.

However I also see a .dll file in use https://github.com/Microsoft/FASTER/blob/master/cs/src/core/...

Which suggests that this won't work in NetCoreApp on linux. It's not crossplatform unless it eliminates this or supplies linux .so binaries and so on.

Re: Faster – Fast key-value store from Microsoft Research

#75

Post by benaadams »

SideburnsOfDoom wrote: post #74
yarper wrote: post #73

Earlier quoted context omitted.

Can anyone work out what platforms it works on? .net core?

Here is the relevant metadata: https://github.com/Microsoft/FASTER/blob/master/cs/src/core/... Should be usable in NetCoreApp 2.0 and up, or .NET full framework 4.6 and up. In other words, yes, it will work on recent versions of NetCore and classic .Net. However I also see a .dll file in use https://github.com/Microsoft/FASTER/blob/master/cs/src/core/... Which suggests that this won't work in NetCoreApp on linux. It'…

src is in native https://github.com/Microsoft/FASTER/tree/master/cs/src/nativ...

Looks like some file IO functions and __rdtsc

Re: Faster – Fast key-value store from Microsoft Research

#76

Post by utopcell »

fizx wrote: post #15

If you want concrete benchmarks, they compare to RocksDB and Redis around page 10 of their academic paper. ( https://www.microsoft.com/en-us/research/uploads/prod/2018/0... ) TL;DR: I find their choice of benchmarks to be very convenient. They tested on in-memory 8 byte payloads and were way faster than RocksDB and Redis. They then tested against only different configurations of themselves for configurations that hit…

8-byte payloads are a necessary limitatuon of their system because of atomic operations.

Re: Faster – Fast key-value store from Microsoft Research

#77

Post by xmichael999 »

DSingularity wrote: post #57
Nican wrote: post #56

I have seen plenty of local-machine fast key-value stores, such as LevelDB (By Google), or RocksDB (By Facebook), but I have a hard time imagining what they are for. What are the use cases for such a library?

Imagine you want to run a service. This service needs to maintain some intermediate state. This state might’ve frequently read. There might be little value in persisting this state. Also, your service is used by many users, so this state can grow to be pretty big. For example, contents of a shopping cart. One solution is to maintain such state in some key-value store. Different functions of your service can query thi…

Great summary.

Re: Faster – Fast key-value store from Microsoft Research

#78

Post by utoku »

brongondwana wrote: post #71
skyde wrote: post #26

I had to opportunity to look at the code. I believe this should be compared against embedded hash database such as "kyoto cabinet, LevelDB, RocksDB" . They introduce a novel latch free hashtable that they say is faster than other in memory data structure. They also introduce a new disk persistence system called HybridLog that combines in-place updates (in memory) and log-structured organization (on disk). The interes…

I was very surprised to see pretty much the entire implementation inside a file called faster.h rather than in a .cc file. Maybe that's all the rage in C++ libraries, I don't work with many of them.

It is because of the template class code. In C++, a template is not really a class or a function.

https://isocpp.org/wiki/faq/templates#templates-defn-vs-decl

Re: Faster – Fast key-value store from Microsoft Research

#79

Post by fahrradflucht »

skyde wrote: post #26

I had to opportunity to look at the code. I believe this should be compared against embedded hash database such as "kyoto cabinet, LevelDB, RocksDB" . They introduce a novel latch free hashtable that they say is faster than other in memory data structure. They also introduce a new disk persistence system called HybridLog that combines in-place updates (in memory) and log-structured organization (on disk). The interes…

To compare it to rocksdb or leveldb: it looks like there is no iteration support or the notion of ordered keys to support partial scans...
Post reply on HN