41–50 of 77 posts
A good filesystem to use as pseudo-db is ZFS. It's designed to have practically limitless number of files, and it's snapshotting/cloning come in really handy for things like backups etc. Also you get full checksumming.
ZFS does have a hard limit of about 280 trillion files in a single directory. It's actually just about the most confining limit it does have.
Earlier quoted context omitted.
It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition. That's crazy. Thanks for that information.
Earlier quoted context omitted.
How is that number calculated? It's less than 2^25 but greater than 2^24 It depends on the filesytem, but it is typically a function of filesystem size (and in ext, the number of block groups and inodes per block group size). To take an example: sudo tune2fs -l | egrep -i 'group' Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 So, we have got 1 inode per 4 blocks. This particular filesystem…
Okay, that makes sense - thanks! I guess there's an upper limit to that number too? It's probably squeezing into a 32 or 64 bit number?
(The exception are empty files, because on most filesystems an empty file does not use any data blocks, but possibly an inode to store file size, permissions, etc.)
Earlier quoted context omitted.
It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
This seems to be one of those bits of old UNIX lore that's being lost - it used to be fairly well-known in sysadmin circles that if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount).
Hopefully not now, this is why we all appreciate people sharing these sorts of information :)
Earlier quoted context omitted.
Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition. That's crazy. Thanks for that information.
Your filesystem's block size is probably 4KiB. So, it's one inode per for blocks. That's crazy. Why? Do you expect to store many files less than 16KiB?
I don't think that amount of information is really suitable for a database? It may be larger in some cases too, so there's not guarantees I can even make about size.
During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I…
git is an example of a program that uses this technique (among others) to store blobs. In case of git, the blob names are already SHA-1 hashes.
Earlier quoted context omitted.
Your filesystem's block size is probably 4KiB. So, it's one inode per for blocks. That's crazy. Why? Do you expect to store many files less than 16KiB?
Each user has some small arbitrary information associated with them, with potentially tens of millions of users (and no doubt more in the future). There's not really a structure due to the nature of the project, each user needs personalised information and structure associated to them. I don't think that amount of information is really suitable for a database? It may be larger in some cases too, so there's not guaran…
During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I…
Now I'm wondering whether that makes for a good design assuming you're using a filesystem that surpasses that limitation. Btrfs?
"If you're working with huge files or a huge number of files, use XFS" went the slogan. I was hit by this recently - with our 500ish (!!) entries in /node_modules and their dependencies of dependencies of dependencies, a standard 8GB VM formatted with ext4 just simply didn't have enough inodes for us to have two copies of our app (deployment swapover). Had to increase the size to 20GB, just to get the extra inodes. X…
You could have also tuned ext4 when you created it with mke2fs. You can specify with -N the max # of inodes desired. Granted, it's not something very many people consider when creating their filesystems; for one thing it's hard to predict how many you will need.