Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread-Per-Core Buffer Management for a modern storage system

vectorized.io

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Thread-Per-Core Buffer Management for a modern storage system

#21

Post by agallego »

mirekrusin wrote: post #19
agallego wrote: post #18

Earlier quoted context omitted.

so redpanda partitions 'raft' groups per kafka partition. so in the `topic/partition` model every partition is it's own raft group (similar to multi raft in cockroachdb). So it is in fact even more important due to the replication cost and therefore the additional work of checksumming, compression, etc. Last, a coordinator core for the one of the TCP connections from a client will likely make requests to remote cores…

Ok, thanks. Does redpanda do some kind of auto anti-affinity on hosts for partition group to spread across remote cores? ps. redpanda link from article is broken, goes to https://vectorized.io/blog/tpc-buffers/vectorized.io/redpand... 404

Oh shoot! thank you... fixing the link give me 5 mins.

So currently the partition allocator - https://github.com/vectorizedio/redpanda/blob/dev/src/v/clus... - is primitive.

But we have a working-not-yet-exposed HTTP admin api on the controller that allows for Out Of Band placement.

so the mechanics are there, but not yet integrated w/ the partition allocator.

Thinking that we integrate w/ k8s more deeply next year.

The thinking at least is that at install we generate some machine labels say in /etc/redpanda/labels.json or smth like that and then the partition allocator can take simple constraints.

I worked on a few schedulers for www.concord.io with Fenzo on top of mesos 6 years ago and this worked nicely for both 'affinity', 'soft affinity' and anti-affinity constraints.

Do you have any thoughts on how you'd like this exposed?

Re: Thread-Per-Core Buffer Management for a modern storage system

#22

Post by wskinner »

bob1029 wrote: post #3

More threads (i.e. shared state) is a huge mistake if you are trying to maintain a storage subsystem with synchronous access semantics. I am starting to think you can handle all storage requests for a single logical node on just one core/thread. I have been pushing 5~10 million JSON-serialized entities to disk per second with a single managed thread in .NET Core (using a Samsung 970 Pro for testing). This includes in…

I learned the same thing while writing a log structured merge tree. Single threaded writes are a must - not only for performance but also simplicity of implementation.

I'm curious what about your use required implementing your own storage subsystem rather than using an embedded key value store like RocksDB.

Re: Thread-Per-Core Buffer Management for a modern storage system

#23

Post by agallego »

carllerche wrote: post #20
dotnwat wrote: post #4

Noah here, developer at Vectorized. Happy to answer any questions.

The article called out 500usec as an upper bound for compute. How do you handle heavier compute operations (TlS, encoding / decoding, ...)

on seastar, you yield a lot. so loops go from

    for( ... : collection) {}
to

   return seastar::do_for_each(collection, callback);

Re: Thread-Per-Core Buffer Management for a modern storage system

#24

Post by jandrewrogers »

wskinner wrote: post #22
bob1029 wrote: post #3

More threads (i.e. shared state) is a huge mistake if you are trying to maintain a storage subsystem with synchronous access semantics. I am starting to think you can handle all storage requests for a single logical node on just one core/thread. I have been pushing 5~10 million JSON-serialized entities to disk per second with a single managed thread in .NET Core (using a Samsung 970 Pro for testing). This includes in…

I learned the same thing while writing a log structured merge tree. Single threaded writes are a must - not only for performance but also simplicity of implementation. I'm curious what about your use required implementing your own storage subsystem rather than using an embedded key value store like RocksDB.

RocksDB has two big limitations that preclude its use for many types of high-performance data infrastructure (which it sounds like the OP's use case was). First, its throughput performance is much worse (integer factor) than what can be achieved with a different design for some applications. Second, it isn't designed to work well for very large storage volumes. Again, easy to remedy if you design your own storage engine or use an alternative one. There are storage engines that will happily drive a petabyte of storage across a large array of NVMe devices at the theoretical limits of the hardware, though not so much in open source.

Another thing to consider is that you lose significant performance in a few different dimensions if your storage I/O scheduler design is not tightly coupled to your execution scheduler design. While it requires writing more code it also eliminates a bunch of rough edges. This alone is the reason many database-y applications write their own storage engines. For people that do it for a living, writing an excellent custom storage engine isn't that onerous.

RocksDB is a fine choice for applications where performance and scale are not paramount or your hardware is limited. On large servers with hefty workloads, you'll probably want to use something else.

Re: Thread-Per-Core Buffer Management for a modern storage system

#25

Post by agallego »

jandrewrogers wrote: post #24
wskinner wrote: post #22

Earlier quoted context omitted.

I learned the same thing while writing a log structured merge tree. Single threaded writes are a must - not only for performance but also simplicity of implementation. I'm curious what about your use required implementing your own storage subsystem rather than using an embedded key value store like RocksDB.

RocksDB has two big limitations that preclude its use for many types of high-performance data infrastructure (which it sounds like the OP's use case was). First, its throughput performance is much worse (integer factor) than what can be achieved with a different design for some applications. Second, it isn't designed to work well for very large storage volumes. Again, easy to remedy if you design your own storage eng…

agreed w/ andrew. rocksdb is pretty heavy. for streaming logs, something much much simpler yields significant performance improvements specially when tied to the IO+CPU priority scheduling.

Re: Thread-Per-Core Buffer Management for a modern storage system

#26

Post by magicalhippo »

wskinner wrote: post #22
bob1029 wrote: post #3

More threads (i.e. shared state) is a huge mistake if you are trying to maintain a storage subsystem with synchronous access semantics. I am starting to think you can handle all storage requests for a single logical node on just one core/thread. I have been pushing 5~10 million JSON-serialized entities to disk per second with a single managed thread in .NET Core (using a Samsung 970 Pro for testing). This includes in…

I learned the same thing while writing a log structured merge tree. Single threaded writes are a must - not only for performance but also simplicity of implementation. I'm curious what about your use required implementing your own storage subsystem rather than using an embedded key value store like RocksDB.

Came to a similar conclusion back in the days when writing a raytracer, and it stopped scaling past 8 or so cores.

Ended up with a system where each thread accumulated results in small buffers, appended pointers to those buffers to a shared "buffer list" which was very fast due to low contention using typical spinlock+mutex combo.

The thread that overflowed the buffer list would then become the single writer by taking on the responsibility to accumulate the results to the shared output image. It would start by swapping in a fresh list, so the other threads could carry on.

The system would self-tune by regulating the size of the shared buffer list so that the other threads could keep working while the one "writer thread" accumulated.

Probably had room for improvement, but after this change it scaled almost linearly to a least 32 cores, which was the largest system available for testing at the time.

The reason for not simply allocating a full output image per thread and accumulate post-render was mainly due to the memory requirements for large output images.

Re: Thread-Per-Core Buffer Management for a modern storage system

#27

Post by nine_k »

agallego wrote: post #25
jandrewrogers wrote: post #24

Earlier quoted context omitted.

RocksDB has two big limitations that preclude its use for many types of high-performance data infrastructure (which it sounds like the OP's use case was). First, its throughput performance is much worse (integer factor) than what can be achieved with a different design for some applications. Second, it isn't designed to work well for very large storage volumes. Again, easy to remedy if you design your own storage eng…

agreed w/ andrew. rocksdb is pretty heavy. for streaming logs, something much much simpler yields significant performance improvements specially when tied to the IO+CPU priority scheduling.

What would be some of your choices?

Re: Thread-Per-Core Buffer Management for a modern storage system

#28

Post by eis »

I'd be interested in the write amplification since Redpanda went pretty low level in the IO layer. How do you guarantee atomic writes when virtually no disk provides guarantees other than on a page level which could result in destroying already written data if a write to the same page fails - at least in theory - and so one has to resort to writing data multiple times.

Re: Thread-Per-Core Buffer Management for a modern storage system

#29

Post by agallego »

nine_k wrote: post #27
agallego wrote: post #25

Earlier quoted context omitted.

agreed w/ andrew. rocksdb is pretty heavy. for streaming logs, something much much simpler yields significant performance improvements specially when tied to the IO+CPU priority scheduling.

What would be some of your choices?

This is a hard question because everything is really dependent on your threading model. One has to start w/ the threading model. At the time I wrote the first line of code in Jan 2019, there wasn't anything that was really amenable to the seastar::future / task-based scheduler with truly async IO (enforced by a reactor stall if greater than 500 micros).... so we wrote our own from scratch .... in fact we wrote it many times over, the first version attempted to use flatbuffers atop my old project - https://github.com/smfrpc/smf but the linearization of buffers proved too costly for long running processes which led to the fragmented buffer approach in the blog post mentioned.
Post reply on HN