Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux: Ext4 data corruption in 6.1.64-1

bugs.debian.org

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Linux: Ext4 data corruption in 6.1.64-1

#121

Post by jcalvinowens »

hedora wrote: post #105
fulafel wrote: post #73

Is there some widely used software that does O_DIRECT writes? MariaDB?

O_DIRECT is almost always the wrong choice. sync_file_range gives you much better control over scheduling of the write backs, and madvise gives you better control over caching policy. There were some old UNIX variants where O_DIRECT actually bypassed the filesystem cache, but Linux's cache is coherent, so reading a file immediately after an O_DIRECT write completes is guaranteed to give the new value. That is more sa…

> but Linux's cache is coherent

Linux achieves that coherency by making O_DIRECT invalidate the page in the page cache. I think it paints a more accurate picture to say the caching is disabled with O_DIRECT, not that it is coherent (although it certainly is).

Re: Linux: Ext4 data corruption in 6.1.64-1

#122

Post by mustache_kimono »

nolist_policy wrote: post #113
GauntletWizard wrote: post #3

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems. ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

Note though that unlike zfs, ext4 has a battle-tested fsck. Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation. To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - tr…

I'm trying to understand what exactly this is testing/demonstrating.

Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency. Doing this could do all sorts of nasty things to data you actually care about.

Re: Linux: Ext4 data corruption in 6.1.64-1

#123

Post by keep_reading »

megous wrote: post #102
dezgeg wrote: post #92

Earlier quoted context omitted.

Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)

Fedora, openSUSE tumbleweed, Manjaro,...

> Fedora

You know it's not that hard to debunk this:

> You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1]

WOW fedora patches! Sounds to me like they're not shipping vanilla.

[1] https://fedoraproject.org/wiki/Kernel

Re: Linux: Ext4 data corruption in 6.1.64-1

#124

Post by keep_reading »

dezgeg wrote: post #92
keep_reading wrote: post #82

Earlier quoted context omitted.

Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels

Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)

Even a single patch to something like default kernel config parameters means it's not vanilla. I'm not arguing that there are significant patches, only that nobody seems to ship a true vanilla kernel.

Re: Linux: Ext4 data corruption in 6.1.64-1

#125

Post by tremon »

LoganDark wrote: post #98
remram wrote: post #94

Earlier quoted context omitted.

I think you're agreeing with userbinator here.

Not really. They're arguing that backups count as recovering from corruption, where my argument is that backups don't count because the only thing you're doing is discarding the corruption, not recovering from it. You're basically throwing the corruption out and getting your data back from somewhere else. So you recover your data, but not from the corruption. You recover it from a backup. Similar, but not the same th…

In this case, I would say that backups are not relevant protection. In order to restore something from backup, you would first need to know that a restore from backup is needed, and secondly you need to know what the last-known good copy is.

This is a silent data corruption bug (data is written at the wrong offset, IIUC) with no known method to detect the error condition except to compare the resulting data with a known-good copy. The next backup run will happily accept the corrupted file contents into the backup vault, and unless someone happens to stumble upon the corrupted data, the last-known good copy will eventually be rotated out.

Re: Linux: Ext4 data corruption in 6.1.64-1

#126

Post by tremon »

Animats wrote: post #6

"Causes non-serious data loss". What does that mean. Only affects cat videos?

I think they meant non-catastrophic (no pun intended). From what I can tell, individual files can get corrupted when opened with O_DIRECT, but the filesystem structure will never fail.

From the link in https://news.ycombinator.com/item?id=38591444 :

> file position is not updated after direct IO write and thus [we] direct IO writes are ending in wrong locations effectively corrupting data.

This may be a major problem for hypervisors using ext4-backed disk images. It wouldn't surprise me if qemu opens image files with O_DIRECT by default, but it would surprise me if professional shops were using ext4 as the backing store rather than a provisioning abstraction layer like dm_thin. Still, anyone using kvm on top of ext4 must not be having a good day.

Re: Linux: Ext4 data corruption in 6.1.64-1

#127

Post by 1000thVisitor »

nolist_policy wrote: post #113
GauntletWizard wrote: post #3

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems. ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

Note though that unlike zfs, ext4 has a battle-tested fsck. Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation. To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - tr…

> To give you an example, fsck.ext4 will happily carve a working filesystem out of random data

What application does this have for recovering user data? Would it not be more interesting to see how much of the user data can be recovered? Or are you implying that the random data can symbolize the user data here and much of it would be recovered?

Re: Linux: Ext4 data corruption in 6.1.64-1

#128

Post by matheusmoreira »

Roark66 wrote: post #40
Grimburger wrote: post #38

Earlier quoted context omitted.

Most data is recoverable albeit slow to do. It takes some very bad conditions or intentional actions to make data recovery impossible. Current high level standards for military/diplomatic data sanatisation are complete physical destruction of hard disks. Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days.

SSDs are not magic. For shredding to be impossible the drives would have to have hidden capacity (and a lot of it, more than 1%). People often say, write levelling makes overwriting useless, but what if you overwrite the entire disk? How would you recover after something like this? Of course providing something like 10% of hidden extra capacity would extend the life of the drive significantly, but are manufacturers r…

> what if you overwrite the entire disk

People who want secure deletion generally want their storage device to remain functional afterwards. Same reason they don't simply destroy the disk to get rid of its contents: those things are expensive. Overwriting the entire SSD every time you want a file gone just isn't a solution.

Re: Linux: Ext4 data corruption in 6.1.64-1

#130

Post by jcalvinowens »

keep_reading wrote: post #124
dezgeg wrote: post #92

Earlier quoted context omitted.

Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)

Even a single patch to something like default kernel config parameters means it's not vanilla. I'm not arguing that there are significant patches, only that nobody seems to ship a true vanilla kernel.

> Even a single patch to something like default kernel config parameters means it's not vanilla.

Anybody who cares that much is very likely already compiling their own kernels (I speak for myself here). It doesn't make sense for distros to do the extra work to support it.

Just expanding on this a bit: my Debian laptop has a Kconfig with modules disabled that only includes the exact set of drivers it needs. It takes ten minutes to rebuild on the laptop when I pull from git. Even if Debian did all the work to let me automatically install the latest vanilla kernel... I'd still build it myself.

Post reply on HN