Unix’s file durability problem
utcc.utoronto.ca
Unix’s file durability problem
1–10 of 161 posts
Re: Unix’s file durability problem
#2Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
Re: Unix’s file durability problem
#3Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
Re: Unix’s file durability problem
#4Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
It only takes one combination to start the chain reaction. If someone identifies one combination of kernel, disk driver, file system, hardware, and syscalls that results in reliable durability then people who care about reliable durability will start using it, and that will eventually turn into a de facto standard which will eventually turn into an actual standard.
Re: Unix’s file durability problem
#5Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
All of it - these things aren't straightforward, you need to go knee-deep in this area of programming to have a good hnderstanding of it.
Re: Unix’s file durability problem
#6Rule of thumb for OS disk I/O: write as soon as possible, i.e. write as soon as possible without hurting performance using buffers on memory for slow component amortization, spreading operations during longer periods.
Re: Unix’s file durability problem
#7Earlier quoted context omitted.
Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
It only takes one combination to start the chain reaction. If someone identifies one combination of kernel, disk driver, file system, hardware, and syscalls that results in reliable durability then people who care about reliable durability will start using it, and that will eventually turn into a de facto standard which will eventually turn into an actual standard.
Until someone who doesn't care changes some code and that pattern is no longer durable.
Re: Unix’s file durability problem
#8A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html
A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().
Re: Unix’s file durability problem
#9I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about.
...backups?
This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepting changes that break this de facto standard'. You might even get this into the Single Unix Specification in some form if you tried hard, because I really do think there's a need here.
Or we could just have everyone perform regular backups like they already are/should be doing, and decide that if systems are crashing so frequently as to lose data often enough, trying to "fix" this "problem" by adding what would likely be another mass of design-by-committee complexity to filesystems is not addressing the cause but only its symptoms.
Then again, with over two decades of experience using the FAT filesystem and never a single instance of unrecoverable data loss despite sudden crashes while hearing countless tales of others corrupting their data frequently even when using far more complex and "robust" filesystems, it makes me wonder why I don't seem to suffer quite the same problems...
Re: Unix’s file durability problem
#10Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
Event assuming that you can look at the source code for your filesystem/kernel the results of a given write still depend on the conditions and orders under which your writes hit disk relative to other processes' writes.
For example, if two processes issue writes to disk sectors that are adjacent but one of the processes' writes also affects a different sector farther away on the physical disk the I/O scheduler may re-order processes' sector level writes. Normally journalling addresses this issue but if the journalling transactions are reordered then it's no help at he application level.