A programmer-friendly I/O abstraction over io_uring and kqueue

Post by dboreham » Thu, Nov 24, 2022, 4:54 AM UTC

p_l wrote: post #17

dboreham wrote: post #14

"The good news is that Windows also has a completion based system similar to io_uring but without batching, called IOCP" fwiw IOCP in NT predates the similar mechanisms in Linux by at least a decade (and the VMS QIO scheme upon which it was in turn based is even older). As I understand it the reason Unix(1) (and then Linux) did not have efficient network I/O kernel interfaces until relatively recently was due to fear…

TL;DR - Async I/O wasn't included in Unices because it's hard & complex and unices are about keeping to the simple till you can no longer lie about it being painless. Considering that Windows NT's IOCP is very close to direct copy of VMS QIO mechanism (and even more underneath in officially undocumented boundary layer between user space and kernel space), I don't think it's a case of patents. UNIX was just always aga…

Post by p_l » Thu, Nov 24, 2022, 7:46 AM UTC

tiffanyh wrote: post #18

p_l wrote: post #17

Earlier quoted context omitted.

TL;DR - Async I/O wasn't included in Unices because it's hard & complex and unices are about keeping to the simple till you can no longer lie about it being painless. Considering that Windows NT's IOCP is very close to direct copy of VMS QIO mechanism (and even more underneath in officially undocumented boundary layer between user space and kernel space), I don't think it's a case of patents. UNIX was just always aga…

> Considering that Windows NT's IOCP is very close to direct copy of VMS QIO mechanism Microsoft hired the main architect of VMS, Dave Cutler, away from DEC to design Windows NT - so this shouldn’t be a surprise. https://news.ycombinator.com/item?id=32602132

Post by eps » Thu, Nov 24, 2022, 7:51 AM UTC

eatonphil wrote: post #6

packetlost wrote: post #3

Earlier quoted context omitted.

Is there any attempt to reorder IO events such that writes (and maybe reads) operate in as contiguous of a manner as possible? It might only be relevant in the case of multiple writes to the same file, so the complexity trade-off might not be worth it for TigerBeetle, but might be worth it for other systems. Edit: also, are you guys hiring? ;)

We are definitely hiring a designer in Europe. :) I asked DJ (not on HN, but hangs out in our community Slack [3] where you can ask further if curious), who knows the disk side of things best, and he responds: The OS is free to reorder writes (this is true for both io_uring and conventional IO). In practice it does this for spinning disks, but not SSDs. The OS is aware of the "geometry" of a spinning disk, i.e. what…

Post by iforgotpassword » Thu, Nov 24, 2022, 8:20 AM UTC

p_l wrote: post #24

dboreham wrote: post #21

Earlier quoted context omitted.

MS fought a legal battle with Dec over Cutler's move. Supposedly that ended up settled such that MS had license to Dec patents and explains why NT resembles VMS yet no lawsuit from Dec. Meanwhile other Unix vendors did not have said license. Fwiw I heard this story directly from Unix vendors OS devs when I worked at Netscape on servers (that supported IOCP on NT and hence often performed much better on a low end PC v…

The legal battle happened because it wasn't just that MS hired Cutler out of Digital - They effectively hired out a team that will as already working on design for next generation OS based on the same principles as VMS, and litigation went down on that IP.

Post by mgoetzke » Thu, Nov 24, 2022, 9:25 AM UTC

valzam wrote: post #26

I just watched your talk at the CMU database talks. Just wanted to say I really appreciate reading/hearing about your approach! TB is a super interesting system, I hope I get to properly use it someday.