21–30 of 53 posts
I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
People consistently underestimate the many ways in which storage can and will fail in the wild. The most vexing storage failure is phantom writes. A disk read returns a "valid" page, just not the last written/fsync-ed version of that page. Reliably detecting this case is very expensive, particularly on large storage volumes, so it is rarely done for storage where performance is paramount.
Not that uncommon failure mode for some SSDs, unclean shutdown is like a dice roll for some of them: maybe you get what you wrote five seconds ago, maybe you get a snapshot of a couple hours ago.
I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
sync; sync; haltI worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
You need to sync twice because Unix is dumb: "According to the standard specification (e.g., POSIX.1-2001), sync() schedules the writes, but may return before the actual writing is done." https://man7.org/linux/man-pages/man2/sync.2.html
I don't know. Now async I/O is all the rage and that is the same idea.
Earlier quoted context omitted.
Not that uncommon failure mode for some SSDs, unclean shutdown is like a dice roll for some of them: maybe you get what you wrote five seconds ago, maybe you get a snapshot of a couple hours ago.
Early SSDs were particularly prone to phantom writes due to firmware bugs. Still have scars from the many creative ways in which early SSDs would routinely fail.
It would suddenly become blank. You have an OS and some data today, and tomorrow you wake up and everything claims it is empty. It would still work, though. You could still install a new OS and keep going, and it would work until next time.
What a friendly surprise on exam week.
Sold it to a friend for really cheap with a warning about what had been happening. It surprise wiped itself for him too.
The OS page cache is not a "problem"; it's a basic feature with well-documented properties that you need to learn if you want to persist data. The writing style seems off in general (e.g. "you're lying to yourself").
AFAIK fsync is the best practice not O_DIRECT + O_DSYNC. The article mentions O_DSYNC in some places and fsync in others which is confusing. You don't need both.
Personally I would prefer to use the filesystem (RAID or ditto) to handle latent sector errors (LSEs) rather than duplicating files at the app level. A case could be made for dual WALs if you don't know or control what filesystem will be used.
Due to the page cache, attempting to verify writes by reading the data back won't verify anything. Maaaybe this will work when using O_DIRECT.
> Link fsync to that write (IOSQE_IO_LINK)
> The fsync's completion queue entry only arrives after the write completes
> Repeat for secondary file
Wait, so the OS can re-order the fsync() to happen before the write request it is supposed to be syncing? Is there a citation or link to some code for that? It seems too ridiculous to be real.
> O_DSYNC: Synchronous writes. Don't return from write() until the data is actually stable on the disk.
If you call fsync() this isn't needed correct? And if you use this, then fsync() isn't needed right?
Earlier quoted context omitted.
You need to sync twice because Unix is dumb: "According to the standard specification (e.g., POSIX.1-2001), sync() schedules the writes, but may return before the actual writing is done." https://man7.org/linux/man-pages/man2/sync.2.html
> Unix is dumb I don't know. Now async I/O is all the rage and that is the same idea.
Earlier quoted context omitted.
You need to sync twice because Unix is dumb: "According to the standard specification (e.g., POSIX.1-2001), sync() schedules the writes, but may return before the actual writing is done." https://man7.org/linux/man-pages/man2/sync.2.html
> Unix is dumb I don't know. Now async I/O is all the rage and that is the same idea.
https://en.wikipedia.org/wiki/Data_Integrity_Field This, along with RAID-1, is probably sufficient to catch the majority of errors. But realize that these are just probabilities - if the failure can happen on the first drive, it can also happen on the second. A merkle tree is commonly used to also protect against these scenarios. Notice that using something like RAID-5 can result in data corruption migrating througho…