Live data from Hacker News

Thread-Per-Core Buffer Management for a modern storage system

vectorized.io

31–34 of 34 posts

Re: Thread-Per-Core Buffer Management for a modern storage system

#31
Disappointed to see that you spent 25% of article space to describe in detail all the ways in which computer hardware got faster, then you promised to show how your project is taking advantage of this, but you are not showing any performance measurements at all. Just a very fancy architecture.

Correct me if I’m wrong, but the only number that I can find is a guarantee that you do not exceed 500 us of latency when handling a request. And it’s not clear if this is a guarantee at all, since you say just that the system will throw a traceback in case of latency spikes.

I would have liked to see the how latency varies under load, how much throughput you can achieve, how the latency long tail looks like on a long-running production load, and comparisons with off-the-shelf systems tuned reasonably.

Re: Thread-Per-Core Buffer Management for a modern storage system

#32
post #31

Disappointed to see that you spent 25% of article space to describe in detail all the ways in which computer hardware got faster, then you promised to show how your project is taking advantage of this, but you are not showing any performance measurements at all. Just a very fancy architecture. Correct me if I’m wrong, but the only number that I can find is a guarantee that you do not exceed 500 us of latency when han…

Hi there! Don't be disappointed :) perf benchmarks coming. The article was about architecture. As a new project we have a lot of content to put out... Stay tuned. Very likely early next month.

Re: Thread-Per-Core Buffer Management for a modern storage system

#33
post #3

More threads (i.e. shared state) is a huge mistake if you are trying to maintain a storage subsystem with synchronous access semantics. I am starting to think you can handle all storage requests for a single logical node on just one core/thread. I have been pushing 5~10 million JSON-serialized entities to disk per second with a single managed thread in .NET Core (using a Samsung 970 Pro for testing). This includes in…

I learned the same thing while writing a log structured merge tree. Single threaded writes are a must - not only for performance but also simplicity of implementation. I'm curious what about your use required implementing your own storage subsystem rather than using an embedded key value store like RocksDB.

Extreme performance (namely, low latency under heavy load) is the principal requirement. I have yet to see anything that can touch my approach, especially under mixed read/write workloads. I am able to write (small, any multithreaded storage architecture.

A secondary requirement is extreme simplicity and safety. My entire implementation is written in managed code and can be understood by a junior developer in one weekend. There is not a single line of code in support of a database feature that we aren't actually going to use.

The final requirement is zero external cost to employ this code. If I own my database implementation, Oracle cannot bill me.

The nice-to-have is being able to follow a breakpoint all the way from user tapping a button down into the b-tree rotation condition logic in the database engine. It also makes profiling performance issues a trivial affair. I like being able to see the actual code in my database engine that is causing a hotpath. This visibility is where additional innovation is possible over time.

Re: Thread-Per-Core Buffer Management for a modern storage system

#34
post #31

Disappointed to see that you spent 25% of article space to describe in detail all the ways in which computer hardware got faster, then you promised to show how your project is taking advantage of this, but you are not showing any performance measurements at all. Just a very fancy architecture. Correct me if I’m wrong, but the only number that I can find is a guarantee that you do not exceed 500 us of latency when han…

Hi there! Don't be disappointed :) perf benchmarks coming. The article was about architecture. As a new project we have a lot of content to put out... Stay tuned. Very likely early next month.

Looking forward to it!
Post reply on HN