Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sparkey – Key/value storage by Spotify

github.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Sparkey – Key/value storage by Spotify

#21

Post by illumen »

blippie wrote: post #7
atdt wrote: post #2

I was surprised to see LevelDB ( https://code.google.com/p/leveldb/ ) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?

I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers. leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.

Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.

Re: Sparkey – Key/value storage by Spotify

#22

Post by rythie »

sp332 wrote: post #20
rythie wrote: post #19

I'm struggling to find something that it does that a webserver pointed at the filesystem doesn't do (with hash-ids for file names). I'm wondering if that's all it is, with a bit of logic to write the files in the correct structure.

From the description: Sparkey is an extremely simple persistent key-value store. You could think of it as a read-only hashtable on disk and you wouldn't be far off.

Good point, I missed that. Read the feature list which lists stuff the filesystem does itself.

Re: Sparkey – Key/value storage by Spotify

#23

Post by krka »

bouk wrote: post #15
woogley wrote: post #14

Earlier quoted context omitted.

The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...

He means the last commit on sparkey

Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.

Re: Sparkey – Key/value storage by Spotify

#24

Post by krka »

There are a bunch of comments related to performance data - the code is available so nothing is stopping anyone from making an unbiased comparison. :)

That said, I intend to publish some sort of performance comparison code / results. The downside with me doing it is that: 1) I know the sparkey code much better than I know level-db or any other solution, so the tuning parameters will probably be suboptimal for the other solutions. 2) I will only focus on our specific usecase (write large bulks, do lots of random reads), which may seem a bit unfair to the most general solutions.

Re: Sparkey – Key/value storage by Spotify

#25

Post by krka »

Here are some preliminary performance benchmarks from my regular workstation (Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz, 8 GB RAM): http://pastebin.com/7buZVgdu

The sparkey usage is fairly optimized, but I just randomly put something together for the level-db, so consider the results extremely biased.

Re: Sparkey – Key/value storage by Spotify

#26

Post by js2 »

huhtenberg wrote: post #13

Looks like a cdb variation that moves index to a separate file and therefore allows changing the database (to a degree) without requiring a rebuild. [0] http://en.wikipedia.org/wiki/Cdb_%28software%29

Indeed, from the README: We used to rely a lot on CDB (which is a really great piece of software). It performed blazingly quick and produces compact files. We only stopped using it when our data started growing close to the 4 GB limit

Re: Sparkey – Key/value storage by Spotify

#27

Post by jdp »

Another cool project is bam[1], a constant key/value server built on similar principles. A single input file (in this case, a TSV file instead of the SPL log file) and an index file. The cool thing about bam is that it uses the CMPH[2] library to generate a minimal perfect hash function over the keys in the input file before putting them in the index file.

[1]: https://github.com/StefanKarpinski/bam [2]: http://cmph.sourceforge.net/

Re: Sparkey – Key/value storage by Spotify

#28

Post by rohansingh »

levosmetalo wrote: post #16

I remember my interview at Spotify where we discussed how to implement thumbnail display service in the most effective way. What we actually came to is something along the lines of this library. I always like it when a company focus on their real problems in job interviews and manage to avoid the brain teaser trap. That way you can have a feeling about the job that you are going to work on there, and see if you reall…

Thanks :-) That is a go-to interview question for us and we actually all do it slightly differently and take it in different directions depending on your expertise or specialization.

It is really closely aligned to what our core service is (distributing and streaming files) and is a great chance to talk with the interviewee and figure out where their strengths are.

Totally encourage other people to interview this way. It's what I've done at the past few companies I've been at and really worked excellently — just think of a problem you're working or have worked on, and distill it into an interview problem.

Re: Sparkey – Key/value storage by Spotify

#29

Post by jflatow »

Similar also to DiscoDB, which does support compression, and uses perfect hashing for constant time-lookup with minimal disk access.

Not only that but it provides lightning-fast conjunctive normal form queries, a.k.a logical combinations of primitive keys. Plus it has Python / Erlang bindings.

http://discodb.rtfd.org

https://github.com/jflatow/discodb

Re: Sparkey – Key/value storage by Spotify

#30

Post by StefanKarpinski »

jdp wrote: post #27

Another cool project is bam[1], a constant key/value server built on similar principles. A single input file (in this case, a TSV file instead of the SPL log file) and an index file. The cool thing about bam is that it uses the CMPH[2] library to generate a minimal perfect hash function over the keys in the input file before putting them in the index file. [1]: https://github.com/StefanKarpinski/bam [2]: http://cmph.…

Wow, didn't expect this to make a mention on the front page of HN today. I never did convince Etsy to let me deploy bam in production, but it's so simple that it should be doable without much fuss. I mainly built it as a proof-of-concept to show that serving static data does not have to be difficult – and that loading large static data sets into a relational database is a truly wasteful, terrible approach. Are you actually using bam "in anger"?
Post reply on HN