Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux: Ext4 data corruption in 6.1.64-1

bugs.debian.org

Post reply on HN
Print view View original

101–110 of 138 posts

Re: Linux: Ext4 data corruption in 6.1.64-1

#101

Post by pjmlp »

smashed wrote: post #74
pjmlp wrote: post #44

The irony of seeing this, after Linus remarks how serious and thorough the file system kernel developers are, versus the other device drivers developers.

https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh... It seems like a commit from the main branch was backported to stable branch, but it actually depended on another one which was not backported. Linux kernel maintainers heavily backport commits as it seems the distinction between bug-fix and new features is hard to do. Also, bug-fix can depend on new features.. I'm always a bit scared when I look at the a…

Still, where is the care that Linus mentioned on the keynote interview?

Re: Linux: Ext4 data corruption in 6.1.64-1

#102

Post by megous »

dezgeg wrote: post #92
keep_reading wrote: post #82

Earlier quoted context omitted.

Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels

Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)

Fedora, openSUSE tumbleweed, Manjaro,...

Re: Linux: Ext4 data corruption in 6.1.64-1

#103

Post by blibble »

smashed wrote: post #74
pjmlp wrote: post #44

The irony of seeing this, after Linus remarks how serious and thorough the file system kernel developers are, versus the other device drivers developers.

https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh... It seems like a commit from the main branch was backported to stable branch, but it actually depended on another one which was not backported. Linux kernel maintainers heavily backport commits as it seems the distinction between bug-fix and new features is hard to do. Also, bug-fix can depend on new features.. I'm always a bit scared when I look at the a…

> I'm wondering who exactly wants to run a stable kernel that receives all these updates so fast. If you want all the latest and greatest, just use -latest?

I find the kernel is now the least stable bit of debian stable

I've had LTS kernels break the networking on my workstation (with a 10 year old onboard NIC) and they deliberately broke ZFS in the past

in LTS there shouldn't be ANYTHING other than security fixes

Re: Linux: Ext4 data corruption in 6.1.64-1

#105

Post by hedora »

fulafel wrote: post #73

Is there some widely used software that does O_DIRECT writes? MariaDB?

O_DIRECT is almost always the wrong choice. sync_file_range gives you much better control over scheduling of the write backs, and madvise gives you better control over caching policy.

There were some old UNIX variants where O_DIRECT actually bypassed the filesystem cache, but Linux's cache is coherent, so reading a file immediately after an O_DIRECT write completes is guaranteed to give the new value. That is more sane than the cache-incoherent approach (how do you know your write is going to a page that is clean in OS cache in the other UNIXes?), but also eliminates most of the code-path-length benefit of O_DIRECT.

Also, if I remember right, O_DIRECT doesn't bypass the I/O scheduler on Linux. That, and bypassing the cache are the two main benefits of the old API, and you get neither.

As a bonus, filesystems like tmpfs passive-aggressively return error if you try to use O_DIRECT.

Re: Linux: Ext4 data corruption in 6.1.64-1

#106

Post by Kunix »

HankB99 wrote: post #85
Kunix wrote: post #50

Have any guidelines been posted somewhere for systems already updated to linux-image-amd64 6.1.64-1?

My desktop and laptop are both root on ZFS so I dodged this bug. (Actually I saw the press on this before upgrading anyway so I'm holding off until I see a fix released.)

I saw the message yesterday late in the evening, I still had 8 hours before my automated updates ran and I thought the package would be revoked by then. Mistake :/

Re: Linux: Ext4 data corruption in 6.1.64-1

#107

Post by ikawe »

To get past apt errors in the immediate (copied from the bottom of the bug report):

> This should block just the buggy kernel. Which might help with unattended upgrades problem or just being forgetful. It might even uninstall the buggy kernel, though I didn't test that. And it shouldn't impact upgrading to 6.1.66 when it's available.

> create a file: /etc/apt/preferences.d/buggy-kernel

> with the contents:

    # avoid kernel with ext4 bug 
    # 1057843
    Package: linux-image-\*
    Pin: version 6.1.64-1
    Pin-Priority: -1
> (the comment isn't required but is helpful for remembering why this file is around in 6 months.)

Re: Linux: Ext4 data corruption in 6.1.64-1

#108

Post by Kunix »

userabchn wrote: post #104

Why wasn't automatic updating to this halted immediately? The "grave" bug report was yesterday, but today unattended-upgrades updated me to the affected kernel and I restarted before I heard about the problem.

I found a few things surprising about this bug too, but this might be due to my own ignorance about how these processes work:

- I was expecting the package to either be revoked, or to have a new `6.1.64-2` version being published with the previously good known state.

Maybe it wasn't done because it was not possible (mirrors being write-only, and maybe other complications in publishing a 6.1.64-2).

- I was expecting some guidelines to be published for affected machines. I've seen questions being asked [1], but no answers to them yet (so users are not sure if it would be safe to rollback to the previous kernel version (6.1.55-1) or not).

Maybe it's because the problem is not well understood yet, or maybe there are not enough people available to answer such questions on a weekend?

If someone can provide some context about "how these things work", it would be interesting to learn more about this.

[^1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843#33

Re: Linux: Ext4 data corruption in 6.1.64-1

#109

Post by WirelessGigabit »

Grimburger wrote: post #43
eh8 wrote: post #39

Earlier quoted context omitted.

> Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days. Which emphasizes the importance of enabling full disk encryption immediately whenever you start using a new device--BitLocker if you're on Windows, FileVault on macOS, LUKS on Linux, etc. Trying to decrypt data is much harder than reconstructing deleted data on a stolen drive.

True, properly zeroing out the headers on an encrypted drive will make recovery impossible. How to do that reliably is another question.

You cannot with normal tools as writing a 0 to the SSD does not guarantee it overwrites the 0 you want. At best it does, at worst it writes the 0 somewhere else and remaps the bit (or whatever its physical storage thing is).

Re: Linux: Ext4 data corruption in 6.1.64-1

#110

Post by WirelessGigabit »

eh8 wrote: post #39
Grimburger wrote: post #38

Earlier quoted context omitted.

Most data is recoverable albeit slow to do. It takes some very bad conditions or intentional actions to make data recovery impossible. Current high level standards for military/diplomatic data sanatisation are complete physical destruction of hard disks. Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days.

> Even linux tools like shred have given up saying they can actually delete data from disks due to how SSD's work these days. Which emphasizes the importance of enabling full disk encryption immediately whenever you start using a new device--BitLocker if you're on Windows, FileVault on macOS, LUKS on Linux, etc. Trying to decrypt data is much harder than reconstructing deleted data on a stolen drive.

This is especially a problem on macOS:

https://support.apple.com/guide/disk-utility/erase-and-refor...

> Note: With a solid-state drive (SSD), secure erase options are not available in Disk Utility. For more security, consider turning on FileVault encryption when you start using your SSD drive.

So if you set up a Mac without FileVault you can never erase everything.

At least with my Lenovo I can do the secure erase.

Post reply on HN