We are working on a platform that lets you measure this stuff with pretty high precision in real time.
You can check out our sandbox here:
IO Devices and Latency
51–60 of 159 posts
Re: IO Devices and Latency
#52Seeing the disk IO animation reminded me of Melvin Kaye[0]:
Mel never wrote time-delay loops, either, even when the balky Flexowriter
required a delay between output characters to work right.
He just located instructions on the drum
so each successive one was just past the read head when it was needed;
the drum had to execute another complete revolution to find the next instruction.
Re: IO Devices and Latency
#53Gosh, this is beautiful. Fantastic work, Ben. <3
Re: IO Devices and Latency
#54Earlier quoted context omitted.
Half on topic: what libs/etc did you use for the animations? Not immediately obvious from the source page. (it's a topic I'm deeply familiar with so I don't have a comment on the content, it looks great on a skim!) - but I've been sketching animations for my own blog and not liked the last few libs I tried. Thanks!
I heavily, heavily abused d3.js to build these.
Small FYI that I couldn't see them in Chrome 133.0.6943.142 on MacOS. Firefox works.
Re: IO Devices and Latency
#55Earlier quoted context omitted.
I heavily, heavily abused d3.js to build these.
Small FYI that I couldn't see them in Chrome 133.0.6943.142 on MacOS. Firefox works.
It's the complete opposite for me — there are no animations in Firefox even with uBlock Origin disabled, but Brave shows them fine.
The browser console spams this link: https://react.dev/errors/418?invariant=418
edit: looks like it's caused by a userstyles extension injecting a dark theme into the page; React doesn't like it and the page silently breaks.
Re: IO Devices and Latency
#56That great infographic at the top illustrates one big reason why 'dev instances in the cloud' is a bad idea.
Re: IO Devices and Latency
#57I think something about distributed storage which is not appreciated in this article: 1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot. 2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and dr…
what do you mean by drains?
Re: IO Devices and Latency
#58Nice blog. There is also a problem that generally cloud storage is "just unusually slow" (this has been noted by others before, but here is a nice summary of the problem http://databasearchitects.blogspot.com/2024/02/ssds-have-bec... ) Having recently added support for storing our incremental indexes in https://github.com/feldera/feldera on S3/object storage (we had NVMe for longer due to obvious performance advantag…
That database architects blog is a great read.
Re: IO Devices and Latency
#59I think something about distributed storage which is not appreciated in this article: 1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot. 2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and dr…
Good points. PlanetScale's durability and reliability are built on replication - MySQL replication - and all the operational software we've written to maintain replication in the face of servers coming and going, network partitions, and all the rest of the weather one faces in the cloud.
Replicated network-attached storage that presents a "local" filesystem API is a powerful way to create durability in a system that doesn't build it in like we have.
Re: IO Devices and Latency
#60I think something about distributed storage which is not appreciated in this article: 1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot. 2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and dr…
what do you mean by drains?
AWS, for one example, provide a feed of upcoming "events" in EC2 in which certain instances will need to be rebooted or terminated entirely due to whatever maintenance they're doing on the physical infrastructure.
If you miss a termination event you miss your chance to copy that data elsewhere. Of course, if you're _always_ copying the data elsewhere, you can rest easy.