Things like this is why I use XFS anywhere I'm not using btrfs, or ZFS.
No Space Left on Device
31–40 of 77 posts
Re: No Space Left on Device
#32Earlier quoted context omitted.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition.
That's crazy. Thanks for that information.
Re: No Space Left on Device
#33Earlier quoted context omitted.
It's a global limit per filesystem. You can check what the limits are for your filesysems, and what the current use is with `df -i`. The thing that git does is for speed.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
That's 6.53%, not 0.0653%. Bit of a difference.
Re: No Space Left on Device
#34"If you're working with huge files or a huge number of files, use XFS" went the slogan. I was hit by this recently - with our 500ish (!!) entries in /node_modules and their dependencies of dependencies of dependencies, a standard 8GB VM formatted with ext4 just simply didn't have enough inodes for us to have two copies of our app (deployment swapover). Had to increase the size to 20GB, just to get the extra inodes. X…
You could have also tuned ext4 when you created it with mke2fs. You can specify with -N the max # of inodes desired. Granted, it's not something very many people consider when creating their filesystems; for one thing it's hard to predict how many you will need.
Also most distro installers don't make it very easy to do this.
Re: No Space Left on Device
#35A good filesystem to use as pseudo-db is ZFS. It's designed to have practically limitless number of files, and it's snapshotting/cloning come in really handy for things like backups etc. Also you get full checksumming.
ZFS does have a hard limit of about 280 trillion files in a single directory. It's actually just about the most confining limit it does have.
Re: No Space Left on Device
#36Earlier quoted context omitted.
It's a global limit per filesystem. You can check what the limits are for your filesysems, and what the current use is with `df -i`. The thing that git does is for speed.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
How is that number calculated? It's less than 2^25 but greater than 2^24
It depends on the filesytem, but it is typically a function of filesystem size (and in ext, the number of block groups and inodes per block group size). To take an example:
sudo tune2fs -l | egrep -i 'group'
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Re: No Space Left on Device
#37Earlier quoted context omitted.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
That's 6.53%, not 0.0653%. Bit of a difference.
Forgetting to times by hundred will eventually kill me, I swear. I'll update it!
Re: No Space Left on Device
#38Earlier quoted context omitted.
It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition. That's crazy. Thanks for that information.
Your filesystem's block size is probably 4KiB. So, it's one inode per for blocks.
That's crazy.
Why? Do you expect to store many files less than 16KiB?
Re: No Space Left on Device
#39Earlier quoted context omitted.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
How is that number calculated? It's less than 2^25 but greater than 2^24 It depends on the filesytem, but it is typically a function of filesystem size (and in ext, the number of block groups and inodes per block group size). To take an example: sudo tune2fs -l | egrep -i 'group' Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 So, we have got 1 inode per 4 blocks. This particular filesystem…
Okay, that makes sense - thanks! I guess there's an upper limit to that number too? It's probably squeezing into a 32 or 64 bit number?
Re: No Space Left on Device
#40Earlier quoted context omitted.
I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressab…
It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
This seems to be one of those bits of old UNIX lore that's being lost - it used to be fairly well-known in sysadmin circles that if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount).