Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixOS on Btrfs+tmpfs

cnx.srht.site

Post reply on HN
Print view View original

41–50 of 55 posts

Re: NixOS on Btrfs+tmpfs

#41

Post by cmurf »

nix23 wrote: post #38
cmurf wrote: post #36

Earlier quoted context omitted.

Without more information on the exact mount error and btrfsck --readonly output, it's a guessing game. In the ordinary cases, btrfs full behavior is the same as other filesystems. It gets full, you can delete files. Keep in mind deleting files on any cow filesystem is a write that consumes free space before space is freed from the delete operation. There is reserve space for this. If you hit an edge case (which would…

>In the ordinary cases, btrfs full behavior is the same as other filesystems. Exactly that's NOT the case: >>-Reboot and end up with unbootable / You can test it for yourself, it "works" reliable since over more then 8 years.

I just tried it like I mentioned in another comment in this thread and cannot reproduce it.

As much effort as you've spent complaining about this problem in this thread you could have provided a proper bug report in the proper venue. It's not going to get fixed by complaining on HN.

So I have to ask if you just want to complain about it or if you want it fixed? Both can be true and legit. But it still requires a report in the proper venue with enough detail for someone else to reproduce.

Re: NixOS on Btrfs+tmpfs

#42

Post by [deleted] »

xorcist wrote: post #27
kilburn wrote: post #15

Earlier quoted context omitted.

> but without the overhead The cost of atime is an extra write every time you read something. Relatime changes this to one atime update per day (by default), low enough that it usually doesn't matter. However, that update per day may have significant impact when you are using Copy-on-Write filesystems (btrfs, zfs). Each time the atime field is updated you are creating a new metadata block for that file. Old blocks ca…

If you don't use atime, there's no need to keep it around. Daily or otherwise. Unless you have reason to believe an application uses atime, it probably doesn't. I've used noatime by default, except for few cases where I know it is used, in professional settings for probably two decades. Hopefully you know what kind of appliciation you are running. There are many parameters in a system and this is just one of them. Th…

[deleted]

Re: NixOS on Btrfs+tmpfs

#43

Post by anotherhue »

Perhaps we can call this a "Considered State" system. No more haphazardly rearranging bytes on your drive. Managed OS objects, linked at boot and your home dir / config dirs under VCS.

I use nixos with zfs on /home, /nix and /persist. Everything else is tmpfs, including /etc. Mostly you can configure applications to read config from /persist, but when not, a bind mount from /etc/whatever to /persist/whatever works pretty well.

I will never use a computer any other way again.

Re: NixOS on Btrfs+tmpfs

#44

Post by cesarb »

> After freeing the new SATA SSD, I also filled it with butter. Yes, all the way, no GPT, no MBR, just Btrfs, whose subvolumes were used in place of partitions

I would not recommend doing that. It might work for now, but there's a high risk of the disk being seen as "empty" (since it has no partition table) by some tool (or even parts of the motherboard firmware), which could lead to data loss. Having an MBR, either the traditional MBR or the "protective MBR" used by GPT, prevents that, since tools which do not understand that particular partition scheme or filesystem would then treat the disk as containing data of an unknown type, instead of being completely empty; and the cost is just a couple of megabytes of wasted disk space, which is a trivial amount at current disk sizes (and btrfs itself probably "wastes" more than that in space reserved for its data structures). Nowadays, I always use GPT, both because of its extra resilience (GPT has a backup copy at the end of the disk) and the MBR limits (both on partition size and the number of possible partition types).

Re: NixOS on Btrfs+tmpfs

#45

Post by mekster »

nix23 wrote: post #9
yjftsjthsd-h wrote: post #2

Neat:) I would never use btrfs myself[0], but very happy to see people exploring all variations of these ideas. The one thing that's starting to bug me though, as I read blog posts about installing nixos: why is the install process so imperative/non-declarative? Once the system is up, the whole thing fits in configuration.nix, but to get there we still have to use masses of shell commands. Is anyone working on bridgi…

I loose every two years (when i test it again) a volume to btrfs, last time 2 month ago, with that simple "trick": -Fill your rootpartionion as root with "dd if=/dev/urandom of=./blabla bs=3m" -rm blabla && sync (we don't want to be unfair to such a fragile system) -Reboot and end up with unbootable / It's a mess, for a filesystem i would declare it as alpha stage.

Why isn't btrfs declared abandoned and just let people move on?

Everytime I read about it, someone is losing data.

Thank god Ubuntu makes zfs very easy to use. No reason to even consider touching btrfs.

Re: NixOS on Btrfs+tmpfs

#46

Post by cmurf »

nix23 wrote: post #39
cmurf wrote: post #35

Earlier quoted context omitted.

I can't reproduce this on a 5.17.5 kernel and loop device. So if you're able to trivially reproduce it, I'm guessing it's configuration specific. It's still a bug, but to find it means providing more detail about that configuration, including the kernel version. Ideally reproduce it with mainline kernel or most recent stable. And post the details on the linux-btrfs list. They are responsive to bug reports. If you dep…

>So if you're able to trivially reproduce it, Make a VM OpenSUSE or Fedora (tested just these two) fill it and see it not boot anymore...it is trivial.

OK this is a very different scenario than what I thought you were talking about (a broken file system). If a file system is full such that writes aren't possible, boot can fail no matter the file system. This isn't a file system problem. It's a failure to manage space.

Most distros require a read-write /sysroot, and expect the ability to write to disk. If they can't write, various services will fail and that can prevent startup from proceeding further. But without any logs, we have no idea what you're actually experiencing.

You are saying it won't boot but that's not at all a case of a broken file system. It's an expected consequence of the file system being full. Since the examples were clear intent of making the file system completely full, it's a setup to prevent the file system from further writes.

Overwriting file systems are expected to run into this problem less, but aren't immune to it. If the data write requirement is new writes, rather than overwriting, it'll fail whether ext4, XFS or Btrfs. If the requirement involves overwriting, it's expected overwriting file systems will succeed where COW file systems simply can't. It might be a valid argument in favor of non-root users being disallowed to use the last 1-5% of free space on any file system.

Re: NixOS on Btrfs+tmpfs

#47

Post by Fnoord »

vladvasiliu wrote: post #23
Fnoord wrote: post #20

Earlier quoted context omitted.

A lot of consumer grade SSDs and flash (microSD, eMMC) don't like it when they are near full. That's why you should set a reserved space and quotas. Notice your dd trick requires root which ignores the reserved space. At some point, its PEBCAK.

> A lot of consumer grade SSDs and flash (microSD, eMMC) don't like it when they are near full I've heard about this, but my understanding was that when this happens, performance becomes extremely poor. While that may be quite bad, it's still worlds apart from losing data. There's also the fact that the user may have partitioned the drive in a such a way to prevent it from ever filling up. Even root can't fill the pa…

> While that may be quite bad, it's still worlds apart from losing data.

Except if you keep overwriting a flash-based storage system, at some point that flash storage gets destroyed (wear level). You can absolutely achieve such by having a near full filesystem on flash. Mechanical harddrives or partitions don't suffer from this issue.

Perhaps the issue occurs more quickly on btrfs, that I don't know, but it could happen on any filesystem. On the other hand, you should have backups. Personally, I use ZFS on two of my machines, with snapshot feature.

Re: NixOS on Btrfs+tmpfs

#48

Post by Fnoord »

nix23 wrote: post #32
Fnoord wrote: post #20

Earlier quoted context omitted.

A lot of consumer grade SSDs and flash (microSD, eMMC) don't like it when they are near full. That's why you should set a reserved space and quotas. Notice your dd trick requires root which ignores the reserved space. At some point, its PEBCAK.

Ah, so it's a special btrfs feature, as root you can kill your filesystem by simply writing a file to it. I wonder what function that serves. But hey how about a quota behind the scenes?....you know like ZFS? AFS? ReFS?...you know so the filesystem tells the user "sorry cant take anymore" before it really cant take anymore? That would be some crazy enterprise level stuff.... You know, a Filesystem that immediately st…

This can happen on any filesystem under the circumstances of it becoming full as root. You are misinformed if you believe such isn't the case.

"Can't boot" is also vague. I've had data loss with XFS in 2002 or so (didn't have backups), couldn't mount filesystem anymore. Thanks to help on IRC from devs I got almost all data back. I've been able to get recover data from a dying Deathstar, too. And then there's the RAID5 write hole (can be mitigated), and RAID5 issues on btrfs (which are well known). For all we know you were using RAID5 shrug. Anyway, did you file a bug report, did you contact the devs?

Re: NixOS on Btrfs+tmpfs

#49

Post by LanternLight83 »

cesarb wrote: post #44

> After freeing the new SATA SSD, I also filled it with butter. Yes, all the way, no GPT, no MBR, just Btrfs, whose subvolumes were used in place of partitions I would not recommend doing that. It might work for now, but there's a high risk of the disk being seen as "empty" (since it has no partition table) by some tool (or even parts of the motherboard firmware), which could lead to data loss. Having an MBR, either…

Neat! I went looking for ways to create a protective MBR, learned a lot from the Gentoo wiki, some interesting info about how Windows does things in the link below, but the way to achieve this seem to be to just format the disk as GPT and then truncate it to the MBR (or just use one big GPT partition).

https://thestarman.pcministry.com/asm/mbr/GPT.htm

Re: NixOS on Btrfs+tmpfs

#50

Post by vladvasiliu »

Fnoord wrote: post #47
vladvasiliu wrote: post #23

Earlier quoted context omitted.

> A lot of consumer grade SSDs and flash (microSD, eMMC) don't like it when they are near full I've heard about this, but my understanding was that when this happens, performance becomes extremely poor. While that may be quite bad, it's still worlds apart from losing data. There's also the fact that the user may have partitioned the drive in a such a way to prevent it from ever filling up. Even root can't fill the pa…

> While that may be quite bad, it's still worlds apart from losing data. Except if you keep overwriting a flash-based storage system, at some point that flash storage gets destroyed (wear level). You can absolutely achieve such by having a near full filesystem on flash. Mechanical harddrives or partitions don't suffer from this issue. Perhaps the issue occurs more quickly on btrfs, that I don't know, but it could hap…

> Except if you keep overwriting a flash-based storage system, at some point that flash storage gets destroyed (wear level). You can absolutely achieve such by having a near full filesystem on flash. Mechanical harddrives or partitions don't suffer from this issue.

Wearing out is yet a different thing. I've had this happen on an SD card. It would refuse to write anything new, although it reported being mostly free. But the stuff that was already on it was readable.

I've had SD cards that got full. They didn't lose any data, and once I'd moved the things off them, they became usable again.

Granted, this was with a digital camera, so using fat32 at the time, so no fancy FS.

Post reply on HN