Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Disk Usage in Linux

ownyourbits.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Understanding Disk Usage in Linux

#11

Post by cmurf »

zepearl wrote: post #9

Questions about "cp --reflink" (I have never used that option so far, but it sounds useful). Quoting "man": "When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails" Q1: this is copy-on-write, right? Q2: once/if the command completes successfully, are there any (potential) dangers (e.g. if I then immediately delete t…

It's essentially duplicating the file metadata into a new inode with shared data extents and incrementing the extent backref count so that the extent tree with those extents shows that there are multiple referrers.

If either file is written to, it's COW.

You can treat both files as if they are normal files, as if they're identical copies. It's just that the duplication doesn't take up additional space other than metadata.

A possible caution, just in general, on Btrfs is that each subvolume gets its own stash of inodes. In a subvolume, inode numbers are unique, you won't see the same inum in use more than once. But inode numbers aren't unique on the volume. So two completely unrelated files can have the same inum but they would be in different subvolumes (including snapshots where this is particularly noticable because a snapshot is really just a pointer to the parent subvolume and all the inums are unchanged.)

Re: Understanding Disk Usage in Linux

#12

Post by cmurf »

simcop2387 wrote: post #10
zepearl wrote: post #9

Questions about "cp --reflink" (I have never used that option so far, but it sounds useful). Quoting "man": "When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails" Q1: this is copy-on-write, right? Q2: once/if the command completes successfully, are there any (potential) dangers (e.g. if I then immediately delete t…

1) Yes, it's copy on write, and requires such support from the underlying filesystem (Btrfs, maybe XFS not sure of what others support it) 2) It's intended to be treated 100% as if it had been copied the classical way. It's not a hardlink or a symlink and can be treated as a completely new file

Support for reflinks in XFS for a while now, although it requires a mkfs time option to enable it. That option was just enabled by default in xfsprogs-5.1.0 which is now in Fedora 31 (still prerelease).

Re: Understanding Disk Usage in Linux

#13

Post by simcop2387 »

cmurf wrote: post #12
simcop2387 wrote: post #10

Earlier quoted context omitted.

1) Yes, it's copy on write, and requires such support from the underlying filesystem (Btrfs, maybe XFS not sure of what others support it) 2) It's intended to be treated 100% as if it had been copied the classical way. It's not a hardlink or a symlink and can be treated as a completely new file

Support for reflinks in XFS for a while now, although it requires a mkfs time option to enable it. That option was just enabled by default in xfsprogs-5.1.0 which is now in Fedora 31 (still prerelease).

Yea my memory was fuzzy and I knew it was planned and being worked on but couldn't remember if it had happened yet. I think F2FS and bcachefs are also supposed to support it but I'm not sure about their level of support either.

Re: Understanding Disk Usage in Linux

#14

Post by zepearl »

cmurf wrote: post #12
simcop2387 wrote: post #10

Earlier quoted context omitted.

1) Yes, it's copy on write, and requires such support from the underlying filesystem (Btrfs, maybe XFS not sure of what others support it) 2) It's intended to be treated 100% as if it had been copied the classical way. It's not a hardlink or a symlink and can be treated as a completely new file

Support for reflinks in XFS for a while now, although it requires a mkfs time option to enable it. That option was just enabled by default in xfsprogs-5.1.0 which is now in Fedora 31 (still prerelease).

Cool => I'll then probably reformat on my root server the partition hosting the images of my VMs from EXT4 to XFS => then to make 100% backups from time to time I would just have to shut them down for a minute, do a "cp --reflink" of their image-file on the host/dom0, then I could start them up again and do anytime later the slow download of their "copied" file.

Any recommendations about a utility (working on a layer higher than the filesystem) that can synchronize only the chunks of a specific file that has changed since the last sync and which generates as final result a "normal" file (no behind-the-curtains database - just copy the identical data from the local/old file and the changes/new data from the remote location to get a normal up-to-date local file)?

Re: Understanding Disk Usage in Linux

#16

Post by cat199 »

zepearl wrote: post #9

Questions about "cp --reflink" (I have never used that option so far, but it sounds useful). Quoting "man": "When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails" Q1: this is copy-on-write, right? Q2: once/if the command completes successfully, are there any (potential) dangers (e.g. if I then immediately delete t…

Q2: (not 100%) this probably wouldn't apply if crossing fs boundaries, and so cloning to another filesystem would probably occupy more space than the source one. Also, many save out files to temporary copies, remove the old file, and move the new one into place. That likely wouldn't track either. Not really 'immediate safety' of the file, but secondary concerns.. Similar caveats apply for traditional soft/hardlinks and sparse files.

Re: Understanding Disk Usage in Linux

#17

Post by cat199 »

ifcho wrote: post #6
chungy wrote: post #5

Earlier quoted context omitted.

First question: Quotas have been supported on Linux for a very long time. All major (and native) file systems support them. Secondly: disk usage accounting for metadata as well as regular file data may or may not be tricky. ZFS always tells you how much data+metadata is used by a file, helped by metadata itself being dynamically allocated on ZFS like everything else is. File systems like ext4 that have fixed metadata…

Project quotas appear to be more similar to cgroups. They are available in xfs and ext4 https://lwn.net/Articles/623835/

The traditional 4.2BSD-style quotas (1983!) on linux also support quotas on unix groups. Not sure if you had this in mind, but anyway.

I suppose project quotas as outlined here would allow multi-group support though.

Another option could be sparsely-provisioned COW LVM volumes.

Re: Understanding Disk Usage in Linux

#18

Post by caf »

zepearl wrote: post #14
cmurf wrote: post #12

Earlier quoted context omitted.

Support for reflinks in XFS for a while now, although it requires a mkfs time option to enable it. That option was just enabled by default in xfsprogs-5.1.0 which is now in Fedora 31 (still prerelease).

Cool => I'll then probably reformat on my root server the partition hosting the images of my VMs from EXT4 to XFS => then to make 100% backups from time to time I would just have to shut them down for a minute, do a "cp --reflink" of their image-file on the host/dom0, then I could start them up again and do anytime later the slow download of their "copied" file. Any recommendations about a utility (working on a layer…

I believe rsync is the utility you are looking for.

Re: Understanding Disk Usage in Linux

#19

Post by minitoar »

lbotos wrote: post #8
minitoar wrote: post #7

Earlier quoted context omitted.

Is that a reference to Hug Bot? https://pbfcomics.com/comics/hug-bot/

Not sure, but I always thought "hug of death" came from reddit: https://en.wikipedia.org/wiki/Slashdot_effect

Ah yes that is surely it.

Re: Understanding Disk Usage in Linux

#20

Post by bscphil »

One feature not mentioned in this article is that BTRFS (and some other file systems) supports transparently compressing files. It's another (very important) reason why the actual number of bits the file contains might not be how much of your disk space the file is actually using.
Post reply on HN