Live data from Hacker News

Sparkey – Key/value storage by Spotify

github.com

11–20 of 57 posts

Re: Sparkey – Key/value storage by Spotify

#11
I wish more projects would follow this kind of readme format, at least somewhat. There are so many new things that popup on HN but have very little information about all the whats and whys I should care

What problem are you solving?

If existing solutions existed what hurdles did you face with them and how did you overcome them with your custom solution?

How do you compare from a performance view? (granted they still need to do this, but at least put in a section about it)

Re: Sparkey – Key/value storage by Spotify

#12
This looks interesting. Can be used for deduplication by keeping this hash table on disk for large amount of data

I wrote something like this to optimize disk seeks heavily by returning a reference of 8 byte and keeping a hashtable in memory. A mostly-append only records store that allowing mutations of same key and by rounding size of blobs by power of 2. Written to optimize storage layer for Membase.

https://github.com/t3rm1n4l/lightkv

Re: Sparkey – Key/value storage by Spotify

#14
post #10
post #2

I was surprised to see LevelDB ( https://code.google.com/p/leveldb/ ) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?

According to GitHub, last commit (which is also initial checkin) is two years old.

The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...

Re: Sparkey – Key/value storage by Spotify

#15
post #14
post #10

Earlier quoted context omitted.

According to GitHub, last commit (which is also initial checkin) is two years old.

The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...

He means the last commit on sparkey

Re: Sparkey – Key/value storage by Spotify

#16
I remember my interview at Spotify where we discussed how to implement thumbnail display service in the most effective way. What we actually came to is something along the lines of this library.

I always like it when a company focus on their real problems in job interviews and manage to avoid the brain teaser trap. That way you can have a feeling about the job that you are going to work on there, and see if you really like both work and people.

Re: Sparkey – Key/value storage by Spotify

#17
post #9
post #3

I'm baffled by the choice of using the GNU autofools chain just to include a Doxygen target in the Makefile. The whole thing is essentially straight up C with just 1 library dependency. The command line argument processing is also quite haphazardly done, it's not like it using getopt or whatever that poses compatibility issues. Is writing and packaging with a Makefile that difficult?

They wrote a database to solve an operational need. From experience I can tell you that's an endeavour you should strife to spend as little time on as possible. I think it's a miracle they produced something they feel comfortable sharing with the world. If you write a database in house, and the tool chain and the argument processing are the only things done haphazardly, then hats off to you :)

Whoa I sense passive-aggressiveness :) I'm still waiting for those benchmarks. The code is very clean and simple I was just picking bones. The use case seems to be overly specific tho. Is there any other example use cases where this library could be useful?

Re: Sparkey – Key/value storage by Spotify

#19
I'm struggling to find something that it does that a webserver pointed at the filesystem doesn't do (with hash-ids for file names). I'm wondering if that's all it is, with a bit of logic to write the files in the correct structure.

Re: Sparkey – Key/value storage by Spotify

#20
post #19

I'm struggling to find something that it does that a webserver pointed at the filesystem doesn't do (with hash-ids for file names). I'm wondering if that's all it is, with a bit of logic to write the files in the correct structure.

From the description: Sparkey is an extremely simple persistent key-value store. You could think of it as a read-only hashtable on disk and you wouldn't be far off.
Post reply on HN