Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HyperLevelDB: A High-Performance LevelDB Fork

hyperdex.org

Post reply on HN
Print view View original

21–30 of 30 posts

Re: HyperLevelDB: A High-Performance LevelDB Fork

#21

Post by monstrado »

halayli wrote: post #20

I am surprised they are using levelDB for a key/value store when they could have used a DBM variant like Kyoto Cabinet HashDB, which is way faster for such a task. Why do you need sorted keys for key/value storage?

Sorted keys are extremely useful for time series data. For example, if a key has a timestamp in it and you'd like to do an aggregation over a few days of "keys"...it's very simple to do an iteration from a STARTKEY to a STOPKEY (the keys in-between don't have to be defined...which is very important). Not only is it simple to do the iteration, it's very fast. This kind of use case (very common) is difficult without sorted keys and iteration capabilities.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#23

Post by rescrv »

Severian wrote: post #22

I'd be interested to see the I/O with the same disk system. Right now the comparison is between a 500GB HD and an SSD. I would assume this would skew results quite a bit.

Please look at the graphs again. Each comparison is done twice: once on hdd and once on ssd. This gives an apples-to-apples comparison for each platform.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#24

Post by halayli »

monstrado wrote: post #21
halayli wrote: post #20

I am surprised they are using levelDB for a key/value store when they could have used a DBM variant like Kyoto Cabinet HashDB, which is way faster for such a task. Why do you need sorted keys for key/value storage?

Sorted keys are extremely useful for time series data. For example, if a key has a timestamp in it and you'd like to do an aggregation over a few days of "keys"...it's very simple to do an iteration from a STARTKEY to a STOPKEY (the keys in-between don't have to be defined...which is very important). Not only is it simple to do the iteration, it's very fast. This kind of use case (very common) is difficult without so…

Agreed - but they are doing a full scan when doing searches so I don't see the benefit.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#25

Post by monstrado »

halayli wrote: post #24
monstrado wrote: post #21

Earlier quoted context omitted.

Sorted keys are extremely useful for time series data. For example, if a key has a timestamp in it and you'd like to do an aggregation over a few days of "keys"...it's very simple to do an iteration from a STARTKEY to a STOPKEY (the keys in-between don't have to be defined...which is very important). Not only is it simple to do the iteration, it's very fast. This kind of use case (very common) is difficult without so…

Agreed - but they are doing a full scan when doing searches so I don't see the benefit.

Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset.

A range iteration allows you to scan a subset of the data, and as long as they didn't fundamentally change how LevelDB works, than they will still support ranges.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#26

Post by halayli »

monstrado wrote: post #25
halayli wrote: post #24

Earlier quoted context omitted.

Agreed - but they are doing a full scan when doing searches so I don't see the benefit.

Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset. A range iteration allows you to scan a subset of the data, and as long as they didn't fundamentally change how LevelDB works, than they will still support ranges.

> Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset.

In this case, you lost the sorted keys advantage that BTree gives you.

My point is that this project leans more towards exact key -> value lookups and a BTree is an overkill here.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#27

Post by monstrado »

halayli wrote: post #26
monstrado wrote: post #25

Earlier quoted context omitted.

Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset. A range iteration allows you to scan a subset of the data, and as long as they didn't fundamentally change how LevelDB works, than they will still support ranges.

> Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset. In this case, you lost the sorted keys advantage that BTree gives you. My point is that this project leans more towards exact key -> value lookups and a BTree is an overkill here.

I guess it depends on your use case, most of the use cases I've seen have been time series using range iterations which is incredibly fast, but I understand your concern if you're only using it for random gets.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#28

Post by halayli »

monstrado wrote: post #27
halayli wrote: post #26

Earlier quoted context omitted.

> Searching is not the same thing as a range iteration, normally if you want to search a Key/Value than you'll need to scan the entire dataset. In this case, you lost the sorted keys advantage that BTree gives you. My point is that this project leans more towards exact key -> value lookups and a BTree is an overkill here.

I guess it depends on your use case, most of the use cases I've seen have been time series using range iterations which is incredibly fast, but I understand your concern if you're only using it for random gets.

Probably providing the option to create a btree or hash map would be nice.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#29

Post by jeffffff »

halayli wrote: post #20

I am surprised they are using levelDB for a key/value store when they could have used a DBM variant like Kyoto Cabinet HashDB, which is way faster for such a task. Why do you need sorted keys for key/value storage?

hash indexes tend to have really terrible write performance because the locations of the writes on disk are random. lsm trees have way better write performance.

Re: HyperLevelDB: A High-Performance LevelDB Fork

#30

Post by _b »

I would like to see a benchmark showing if HyperLevelDB's changes made compaction slower or faster. (This is important because LevelDB's bottleneck is commonly compaction.)

Under the compaction section, a chart does show a dramatic improvement on the fillrand benchmark. But since HyperLevelDB removes the LevelDB write delay when compaction falls behind, writes in HyperLevelDB will continue at full speed even as compaction falls extremely far behind. So, yes, the data for the benchmark was technically written into the db, but at the costs of making reads insanely expensive.

Most reads need to touch every level-0 file. If compaction moving data out of level-0 fell 10GB behind, then there would be hundreds of level-0 files, and most reads would need to check all of them.

If the benchmark had mixed in one random read for every random write, I assume HyperLevelDB would have been dramatically slower than LevelDB.

But if the changes in what to pick for compaction made HyperLevelDB non-trivially faster, that would be quite interesting. But I can't tell if they did.

Post reply on HN