Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No Space Left on Device

carlmastrangelo.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: No Space Left on Device

#51

Post by microtonal »

dvcrn wrote: post #16
JonRB wrote: post #4

During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I…

inodes are by the way a great topic to talk about in a job interview. People working with unix based systems should know about them, or at least encountered them once in their career. Another good one that even less people know about is the sticky bit.

Indeed. One of the types questions were people typically bomb (this is also one in Google's telephone interview arsenal): where is a filename stored?

People who have the vague notion of 'an inode stores metadata' will usually answer this incorrectly.

Re: No Space Left on Device

#52

Post by caf »

Xylakant wrote: post #21
rshm wrote: post #6

No date on the post. Probably old and fixed now. Just checked with a random folder, it happily supported 782292 files.

Since the root cause is a hash collision in the dir_index code you might just have been lucky and the author of the article unlucky.

They would have had to have been exceptionally lucky - the chance of getting no collisions in a 32 bit hash among 782292 inputs is about 10^-62.

Re: No Space Left on Device

#53

Post by rshm »

Xylakant wrote: post #21
rshm wrote: post #6

No date on the post. Probably old and fixed now. Just checked with a random folder, it happily supported 782292 files.

Since the root cause is a hash collision in the dir_index code you might just have been lucky and the author of the article unlucky.

Probably lucky. File names are md5 of the content, i am not sure if it helped on collision on 32 bit md4.

The linked post mentions dir_index default and tune2fs -l lists the dir_index under Filesystem features only.

Not knowing this default behavior, i was considering enabling dir_index explicitly to aid chocking rsync.

EDIT: I ran the reproduction code with 200K instead of 100K just to be sure. There was no problem creating file as well as copying all the files to a different folder. The disk was formatted about 3 months ago and mounted with default options "ext4 noatime,barrier=0 0 0" . On original as well as copied folder `ls | wc -l` still gives 200000.

Re: No Space Left on Device

#55

Post by majewsky »

oxplot wrote: post #47
JonRB wrote: post #4

During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I…

Standard practice to store a large number of files but in such a way that they don't end up in a single directory, is to create a crypto hash of your object names and use the first n characters as the directory name. Properties of hash functions will ensure that the files are distributed evenly between all the directories. git is an example of a program that uses this technique (among others) to store blobs. In case…

You don't actually need a cryptographic hash here [1]. I'd rather use a fast hash like Murmur instead.

[1] Exception: I don't know how prone Murmur is to hash collisions, so if collision-based attacks are part of your threat model, a randomly salted cryptographic hash might be appropriate.

Re: No Space Left on Device

#56

Post by pjc50 »

cyberferret wrote: post #22

Brings back memories of the old DOS days, when there was quite a low limit on the number of files that you could store in the root directory. I can't remember the exact numbers, but it lead to my early habits of creating a new sub directory for every little thing so nowadays (over 3 decades later) nearly everything I need is at least 5 levels deep! ;) NB: My current working folders are now in Google Drive, but is ess…

.. aided by CHKDSK, which would find orphaned allocation units and create files for them in the root directory. Could quite easily fill up your root directory on its own.

Re: No Space Left on Device

#58

Post by majewsky »

microtonal wrote: post #43
bArray wrote: post #39

Earlier quoted context omitted.

Okay, that makes sense - thanks! I guess there's an upper limit to that number too? It's probably squeezing into a 32 or 64 bit number?

There is probably a upper limit, but practically you don't want more inodes than blocks. Since a non-empty file at least takes up one block, assuming that your files are all at least 1 byte, you couldn't use more inodes than blocks anyway. (The exception are empty files, because on most filesystems an empty file does not use any data blocks, but possibly an inode to store file size, permissions, etc.)

If I'm not mistaken, some file systems store file contents smaller than a block directly inside the respective directory entry.

Re: No Space Left on Device

#59

Post by ajuc »

I've had similar error in production. We had linux server with C++ program doing many things, and in the process it created temporary files for a barcode printer. The files were named uniquely, the server had lots of free space, we left them there to allow easier debugging if sth wrong happened (it happened a few times each year - we created the labels to print by glueing escape codes and sometimes there were labels that didn't worked for one reason or another), and only cleaned the old files when there was not much space left.

At some point there was a bug when sometimes users got "wrong permissions or no space left" errors (sorry don't remember the exact error and IIRC our app just tried to save a file and failed with unspecific error if it couldn't write the new temporary file, so we haven't had the exact system error).

What was weird is - there was still lots of space on that disk, permissions were OK, and I've tried creating new files there, both from our app and by hand and it worked.

It turned out after a few days of frantic debugging, that when there's too many files with similar names in one directory - creating a new file can fail because of hash conflict, depending on the filename only. So,

    ls A1234.txt S1234.txt # no results
    touch S12345.txt   # fails
    touch S1234.txt   # works
:) It was one of the biggest WTF moments I had while programming.
Post reply on HN