A programmer-friendly I/O abstraction over io_uring and kqueue
1–10 of 34 posts
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#2Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#3Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? It might only be relevant in the case of multiple writes to the same file, so the complexity trade-off might not be worth it for TigerBeetle, but might be worth it for other systems.
Edit: also, are you guys hiring? ;)
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#4Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? It might only be relevant in the case of multiple writes to the same file, so the complexity trade-off might not be worth it for TigerBeetle, but might be worth it for other systems. Edit: also, are you guys hiring? ;)
> Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible?
Something like a disk elevator at a higher level of the stack?
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#5Earlier quoted context omitted.
Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? It might only be relevant in the case of multiple writes to the same file, so the complexity trade-off might not be worth it for TigerBeetle, but might be worth it for other systems. Edit: also, are you guys hiring? ;)
> Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? Something like a disk elevator at a higher level of the stack?
Yeah, even tweaking kernel IO scheduling would probably be sufficient. It'll depend on spinning rust vs SSDs though.
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#6Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? It might only be relevant in the case of multiple writes to the same file, so the complexity trade-off might not be worth it for TigerBeetle, but might be worth it for other systems. Edit: also, are you guys hiring? ;)
We are definitely hiring a designer in Europe. :)
I asked DJ (not on HN, but hangs out in our community Slack [3] where you can ask further if curious), who knows the disk side of things best, and he responds:
The OS is free to reorder writes (this is true for both io_uring and conventional IO).
In practice it does this for spinning disks, but not SSDs.
The OS is aware of the "geometry" of a spinning disk, i.e. what sectors are physically close to each other.
But for NVME SSDs it is typically handled in the firmware. SSDs internally remap "logical" addresses (i.e. the address from the OS point of view) to "physical" addresses (actual locations on the SSD).
e.g. if the application (or OS) writes to block address "1" then "2", the SSD does not necessarily store these in adjacent physical locations. (OSTEP explains this well [0].)
"Performance Analysis of NVMe SSDs and their Implication on Real World Databases" explains in more detail:
> In the conventional SATA I/O path, an I/O request arriving at the block layer will first be inserted into a request queue (Elevator). The Elevator would then reorder and combine multiple requests into sequential requests. While reordering was needed in HDDs because of their slow random access characteristics, it became redundant in SSDs where random access latencies are almost the same as sequential. Indeed, the most commonly used Elevator scheduler for SSDs is the noop scheduler (Rice 2013), which implements a simple First-In-First-Out (FIFO) policy without any reordering.
Applications can help performance by grouping writes according to time-of-death (per "The Unwritten Contract of Solid State Drives" [2]), but the SSD is free to do whatever. We are shortly going to be reworking the LSM's compaction scheduling to take advantage of this: https://github.com/tigerbeetledb/tigerbeetle/issues/269.
[0] https://pages.cs.wisc.edu/~remzi/OSTEP/file-ssd.pdf
[1] https://www.cs.binghamton.edu/~tameesh/pubs/systor2015.pdf
[2] https://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf
[3] https://join.slack.com/t/tigerbeetle/shared_invite/zt-1gf3qn...
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#7Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Can this benefit BSDs?
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#8Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Can this benefit BSDs?
I think FreeBSD invented kqueue and from a quick search it looks like OpenBSD and NetBSD also adopted it. We've seen at least one person slightly tweak TigerBeetle to run on FreeBSD already through the darwin code paths.
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#9Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Is there a reason why you’re not using libdispatch on darwin and instead using kqueue directly?
Re: A programmer-friendly I/O abstraction over io_uring and kqueue
#10Hey folks! Phil from TigerBeetle here. Happy to answer questions or pull in King when I cannot. :)
Is there a reason why you’re not using libdispatch on darwin and instead using kqueue directly?
At the time, we wanted to get macOS or Darwin running as soon as possible to improve the local developer experience, but—we've always had a soft spot for FreeBSD so kqueue was two birds in one!