Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can list a directory containing 8 million files But not with ls..

olark.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: You can list a directory containing 8 million files But not with ls..

#11

Post by bcx »

rarrrrrr wrote: post #10

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

In this case since it was a virtualized disk and it was reading in 32K chunks, I am fairly confident that this wouldn't have helped.

Certainly find . would have been faster without calling stat().

Does os.listdir() stat?

Re: You can list a directory containing 8 million files But not with ls..

#12

Post by justincormack »

The man pages really do try to put you off using the syscalls. I only used getdents for the first time when I was writing a syscall interface for Lua https://github.com/justincormack/ljsyscall - not tried it on 8m files but you can set the buffer size so it should work.

The native aio interface has advantages over the posix wrapper too, and some of the other interfaces. On the other hand the futex interface is not nice to use, as it requires architecture specific assembly which seems undocumented...

Re: You can list a directory containing 8 million files But not with ls..

#13

Post by malkia »

In a similar fashion, something that is reading directories as files on Windows: https://gist.github.com/1148070 - it allows me to get faster some things (it's very crude, rough code, written for exploration, rather than real usage).

Also to get the NTFS streams (metadata).

http://msdn.microsoft.com/en-us/library/aa364226(v=vs.85).as...

Re: You can list a directory containing 8 million files But not with ls..

#14

Post by jesboat »

rarrrrrr wrote: post #10

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

H tried using find, but, in this case, the libc readdir function was the bottleneck, so find doesn't help.

Often, when ls is being slow, you can speed things up drastically by disabling the things that require a stat (colors, -F, etc.) that are often added by distro's shell files (invoking ls with a full path is an easy way to disable shell aliases.) Also, sorting is on by default, which slows things for obvious reasons.

When all you need to do is kill the stat overhead for small dirs on slow file systems, "echo *" is beautifully simple.

Re: You can list a directory containing 8 million files But not with ls..

#15

Post by moe »

rarrrrrr wrote: post #10

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

Actually 'find' will also stat each entry no matter what.

Many of the standard-tools that most people would intuitively expect to be rather optimized (find, rsync, gzip) are embarrassingly inefficient under the hood and turn belly up when confronted with data of any significant size.

That probably stems from the fact that most of the development on these tools took place during a time when 1GB harddrives were "huge" and SMP was "high end".

Re: You can list a directory containing 8 million files But not with ls..

#16

Post by haberman »

rarrrrrr wrote: post #10

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

The article discusses a bottleneck in readdir(), not stat(). Running your command has the same problem as running ls:

    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
    fcntl(5, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
    fchdir(5)                               = 0
    getdents(5, /* 2 entries */, 32768)     = 48
    getdents(5, /* 0 entries */, 32768)     = 0
    close(5)                                = 0
It's only reading 32k at a time, but the author had 500MB of data to be fetched.

Re: You can list a directory containing 8 million files But not with ls..

#17

Post by kylek »

I ran into this problem before. One of our engineers had a script that kept some temporary data in his home directory, which went crazy at one point and generated millions of files (no clue the reasoning for this). Anyways, this was HELL on our backup solution, which ended up failing several nights in a row on that area. Luckily, since the files were largely useless, this left our options open. I think the kicker was the 'rm' command not working (same issue as listing? This was a Solaris 8 system I think). I believe we ended up moving all of the other data off of that filesystem and newfs'ing it.

Re: You can list a directory containing 8 million files But not with ls..

#20

Post by uxp »

carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen.

Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

Post reply on HN