Linux 5.10 BTRFS performance regression

Oy vey, veering off topic, but whatever...

BAM is a sequence of zlib-compressed blocks. Each of them has a checksum, and any corruption gets past it with probability 2^-32. Any sane software will give you an error. When in doubt, just decompress the stupid file using gunzip or zcat, those will verify the checksum. Don't guess, check.

But if you insist on using software that just ignores the checksum (I think, not even samtools, which segfaults(!) on misspelled file names(!) is that awful), all bets are off. Decompressing a corrupted file yields nonsense that doesn't even have the same length as the original data. BAM would also quickly get out of sync, and subsequent records would contain complete nonsense. You'd know if you saw that.

However...

> you are positive that BAM files should be self consistent?

That is a loaded question. In addition to being syntactically correct, BAM files should maintain a few variants that are hard to check. They rearely do. BWA(!) produces inconsistent BAM files, and I've seen many, many scripts in the wild that produce complete bullshit that only happens to work in a very specific setting. Like I said, you're probably using some awful software.

I work (well, until January) for a company that specializes in Ceph support. And it is depressing to see how many customers have crashing OSDs or MDSs, or unexplainabe memory overuse problems. It looks much worse than btrfs. I don't want this at home. But, to be fair, for every distributed storage system (even as simple as DRBD), there is be a person who had extremely negative experience with it.

I wasn't involved so can't personally provide details of how this was done at Sun. Most of my knowledge comes from listening to talks and reading books on ZFS.

For work I'm involved in relating to safety-critical systems, we use the V-model for concepts, requirements, design and implementation, with extensive validation and verification activities at each level. Tools are used to manage all of the requirements, design details and implementation details and link them all together in a manner which aims to require self-consistency at all levels. When done correctly, this means that the person writing the code does not need to be particularly creative at this stage: the structure is completely detailed by the formal design. But it does require significant up-front effort to carefully consider and nail down the design to this level of detail. But it does avoid the need to continually revise and adapt an incomplete or bad design in a never-ending implementation phase.

This approach is definitely not for everyone, and there are many things one can criticise about it. But if you are willing to bear the financial cost and time costs of doing that detailed design work up front, the cost of implementation will be much lower and the product quality will be much greater. There is a lot to be said for not madly mashing keys and churning out code without thinking about the big picture, and Btrfs is a case study in what not to do.

This is just my own personal take on things; I'd definitely recommend reading up on the differences between Waterfall, Agile and the V-model (and Spiral model). Note that you'll see it said that the V-model is based upon Waterfall, which is somewhat true, but it's not necessarily incompatible with Agile. You can combine the two and go all the way down and back up the "V" in sprints or "product increments", but you do need the resources to do all the revalidation and reverification at all levels each time, and this can be costly (this is effectively what the Spiral model is).

In terms of deciding if meticulous up-front design is necessary (again my own take), it depends upon the consequences of failure in the requirements, specifications, design and/or implementation. A random webapp doesn't really have much in the way of consequences other than a bit of annoyance and inconvenience. A safety-critical system can physically harm one or multiple people. Examples: car braking systems, insulin pumps, medical diagnostics, medical instruments, elevator safety controls, avionics etc. It also depends upon how feasible it is to upgrade in the field. A webapp can be updated and reloaded trivially. An embedded application in a hardware device is not trivial to upgrade, especially when it's safety-critical and has to be revalidated for the specific hardware revision.

For filesystems the safety aspect will relate to maintaining the integrity of the data you have entrusted to its care. Computer software and operating systems can have all sorts of silly bugs, but filesystem data integrity is one place where safety is sacrosanct. We set a high bar in our expectation for filesystems, not unreasonably, and after suffering from multiple dataloss incidents with Btrfs, it's clear their work did not meet our expectations. We're not even going into the performance problems here, just the data integrity aspects.

I can't say anything about the tools I use in my company. There are specialist proprietary tools available to help with some of the requirements and specifications management. I will say this: the tools themselves aren't really that important, they are just aids for convenience. The regulatory bodies don't care what tools you use. The important part is the process, of having detailed review at every level before proceeding to the next, and the same again when it comes to validation and verification activities.

Often open source projects limit themselves to some level of unit testing and integration testing, which is fine. But the coverage and quality of that testing may leave some room for improvement. It's clear that Btrfs didn't really test the failure and recovery codepaths properly during its development. Where was the individual unit testing and integration test case coverage for each failure scenario? Where the V-model goes above and beyond this is in the testing of the basic requirements and high-level concepts themselves. You've got to check that the fundamental premises the software implementation is based upon are sound and consistent.

n=1 is n=None ;)