Live data from Hacker News

Linux: Ext4 data corruption in 6.1.64-1

bugs.debian.org

111–120 of 138 posts

Re: Linux: Ext4 data corruption in 6.1.64-1

#111
post #50

Have any guidelines been posted somewhere for systems already updated to linux-image-amd64 6.1.64-1?

I also manually upgraded yesterday (bad luck, just a few hours after the update) and just now reverted to the previous kernel. Not sure this was the right thing to do but didn't want to take any chances

Re: Linux: Ext4 data corruption in 6.1.64-1

#112
post #88

Here's my understanding so far: In the upstream Linux kernel there were two fixes posted months from each other, one for direct io [0] and the other one for ext4 [1]. The ext4 one was marked for backport to stable (CC: stable@vger.kernel.org), the other was not. The problem is that these commits depend on each other for things to work properly. If you have both, you're fine. If you have only the backported one, you h…

Were any Ubuntu versions affected?

Re: Linux: Ext4 data corruption in 6.1.64-1

#113

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems. ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

Note though that unlike zfs, ext4 has a battle-tested fsck.

Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation.

To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself:

  # create a working filesystem image
  dd if=/dev/zero of=test1.img bs=1M count=256
  /sbin/mkfs.ext4 test1.img
  # write file with random data
  dd if=/dev/urandom of=test2.img bs=1M count=256
  # copy superblock into random file
  dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc
  /sbin/fsck.ext4 -fy test2.img
  sudo mount test2.img /mnt/somewhere

Re: Linux: Ext4 data corruption in 6.1.64-1

#114

Why wasn't automatic updating to this halted immediately? The "grave" bug report was yesterday, but today unattended-upgrades updated me to the affected kernel and I restarted before I heard about the problem.

The way the Debian update process works doesn't make it possible to halt updates immediately. The servers you're getting the updates from aren't controlled by the Debian project itself, but are operated by third parties. They sync their data from a master machine (which is operated by Debian) 4 times per day. If you're using a mirror that's not under the debian.org domain, there might even be another machine or two in between, each adding some delay. So even if Debian pulled the update immediately, which their archive software isn't really setup to do, it'd take a bit for that to propgate through to the mirrors.

If this all seems archaic, keep in mind that all this has been designed 20+ years ago, when bandwidth was quite a bit less abundant.

Re: Linux: Ext4 data corruption in 6.1.64-1

#115
post #108

Why wasn't automatic updating to this halted immediately? The "grave" bug report was yesterday, but today unattended-upgrades updated me to the affected kernel and I restarted before I heard about the problem.

I found a few things surprising about this bug too, but this might be due to my own ignorance about how these processes work: - I was expecting the package to either be revoked, or to have a new `6.1.64-2` version being published with the previously good known state. Maybe it wasn't done because it was not possible (mirrors being write-only, and maybe other complications in publishing a 6.1.64-2). - I was expecting s…

The `6.1.66-1` release with a fix was uploaded to Debian's infrastructure about 2 hours after the bug was filed, but it takes a bit for it to be compiled for all Debian's architectures (the mipsel builder e.g. needed 9 hours) and for it to propgate through the mirror network.

Re: Linux: Ext4 data corruption in 6.1.64-1

#116
post #95

Earlier quoted context omitted.

If they are, then that's why the quality is declining.

Funny. You don't even know if that's the case, but somehow you know it's the cause?

Weird. I had a few upvotes from people with a sense of humor before it went back down to 1 and these comments appeared.

Re: Linux: Ext4 data corruption in 6.1.64-1

#117
post #9
post #7

In Debian bookworm: "sudo systemctl stop unattended-upgrades.service" ... wasn't able to prevent unattended-upgrades from going on ahead and just upgrading (to this problematic kernel) anyway. Unintuitively, the "right" way to disable unattended-upgrades is: "sudo dpkg-reconfigure unattended-upgrades" ...and choose "No" when asked.

Most likley because you stopped the wrong service. You should have stopped the relevant timer service, not the service that the timer starts.

> You should have stopped the relevant timer unit, not the service unit that the timer starts.

FTFY. Helps keeping them apart to not use the same word for both ;)

    systemctl disable --now unattended-upgrades.timer;
    systemctl disable --now unattended-upgrades.service
No need to uninstall or mask.

Re: Linux: Ext4 data corruption in 6.1.64-1

#118
post #88

Here's my understanding so far: In the upstream Linux kernel there were two fixes posted months from each other, one for direct io [0] and the other one for ext4 [1]. The ext4 one was marked for backport to stable (CC: stable@vger.kernel.org), the other was not. The problem is that these commits depend on each other for things to work properly. If you have both, you're fine. If you have only the backported one, you h…

Were any Ubuntu versions affected?

I think not unless you've explicitly installed the 6.1 series (none of the official supported releases ship with it by default as far as I can tell).

Re: Linux: Ext4 data corruption in 6.1.64-1

#119
post #93
post #7

In Debian bookworm: "sudo systemctl stop unattended-upgrades.service" ... wasn't able to prevent unattended-upgrades from going on ahead and just upgrading (to this problematic kernel) anyway. Unintuitively, the "right" way to disable unattended-upgrades is: "sudo dpkg-reconfigure unattended-upgrades" ...and choose "No" when asked.

systemctl stop is a oneshot operation, right? Perhaps sudo apt remove unattended-upgrades?

[deleted]

Re: Linux: Ext4 data corruption in 6.1.64-1

#120
post #88

Here's my understanding so far: In the upstream Linux kernel there were two fixes posted months from each other, one for direct io [0] and the other one for ext4 [1]. The ext4 one was marked for backport to stable (CC: stable@vger.kernel.org), the other was not. The problem is that these commits depend on each other for things to work properly. If you have both, you're fine. If you have only the backported one, you h…

6.1.65 is affected as well.
Post reply on HN