Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can list a directory containing 8 million files But not with ls..

olark.com

Post reply on HN
Print view View original

71–80 of 93 posts

Re: You can list a directory containing 8 million files But not with ls..

#71

Post by bmurphy »

qntm wrote: post #68
uxp wrote: post #20

Earlier quoted context omitted.

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

Speaking as a novice Ubuntu user, I don't even know what "install a port" means. Do you mean "open a port"?

Nope, he does not mean open a port. Ports is the name of the packaging system FreeBSD uses, and a port is the equivalent (more or less) of an RPM or deb source package.

http://www.freebsd.org/ports/

Re: You can list a directory containing 8 million files But not with ls..

#72

Post by adestefan »

nodata wrote: post #57
tzs wrote: post #31

I suspect the author is incorrect in his claim that reading in 32k chunks is responsible for the slowness. Due to read ahead and buffering, Unix-like systems tend to do reasonably well on small reads. Yes, big reads are better, but small reads are not unreasonably slow. To test this, he should try "ls | cat". On large directories that often runs many orders of magnitude faster than "ls". This is because, I believe, l…

> To test this, he should try "ls | cat". Running /bin/ls will bypass the alias.

Prefixing the command with a \ will also disable the alias (e.g. \ls)

Re: You can list a directory containing 8 million files But not with ls..

#74

Post by joblessjunkie »

mml wrote: post #73

What about 'echo *' ? That's usually my last resort when dealing with malfunctioning ls.

The shell will attempt to expand '*' into arguments before it spawns the 'echo' process.

The shell cannot handle long argument lists, so this will fail rather quickly.

Re: You can list a directory containing 8 million files But not with ls..

#75

Post by jamesgeck0 »

uxp wrote: post #20
carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

Arch has a Ports-inspired system called Pkgbuild. Given the level of competence the distro expects of the user to start with, I doubt most Arch users would have a great deal of trouble adapting to FreeBSD.

Re: You can list a directory containing 8 million files But not with ls..

#76

Post by smackfu »

joblessjunkie wrote: post #74
mml wrote: post #73

What about 'echo *' ? That's usually my last resort when dealing with malfunctioning ls.

The shell will attempt to expand '*' into arguments before it spawns the 'echo' process. The shell cannot handle long argument lists, so this will fail rather quickly.

Yep, this is usually the first point of failure for commands acting on somewhat large directories, that drives people to use find and exec.

Re: You can list a directory containing 8 million files But not with ls..

#77

Post by shabble »

drhodes wrote: post #28

getdents stands for "get directory entries", in case anyone else was wondering.

Hark back to the days when 8 character function names were a luxury!

And remember to put all your variable declarations at the top of each block, so the compiler can handle it all in a single pass :)

Re: You can list a directory containing 8 million files But not with ls..

#78

Post by alyandon »

tzs wrote: post #31

I suspect the author is incorrect in his claim that reading in 32k chunks is responsible for the slowness. Due to read ahead and buffering, Unix-like systems tend to do reasonably well on small reads. Yes, big reads are better, but small reads are not unreasonably slow. To test this, he should try "ls | cat". On large directories that often runs many orders of magnitude faster than "ls". This is because, I believe, l…

I just tried this on non-virtualized hardware with 10 million files in a single directory using the zfsonlinux native zfs implementation for linux. It took a little over 4 minutes to do a "\ls -f | wc -l" so this might very well be something to do with virtualization.

I'll try an ext3 file system just for giggles and post the results.

Edit:

Didn't have an ext3 file system with enough free inodes handy so I used an ext4 file system. It takes too long to create 10 million files so I cut the test off early. It took about 7 seconds to complete a "\ls -f | wc -l" with 6.2 million files in a single directory.

Re: You can list a directory containing 8 million files But not with ls..

#79

Post by lloeki »

uxp wrote: post #20
carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

Even when there's a compilation step it can be hidden. Homebrew on OS X does compile each package, but produces no actual compiler output. VMWare Player regularly recompiles kernel modules when it detects a kernel upgrade invalidating them. To the user it looks like an bullet list installation step.

Re: You can list a directory containing 8 million files But not with ls..

#80

Post by unwind »

unwind wrote: post #62
scott_s wrote: post #2

Excellent writeup. Computer systems are discoverable. That attitude, along with some of the basic tools (such as strace, ltrace, man pages, debuggers and a compiler) and a willingness to dive into system source code go a long way. If your tracing leads you to a library call and you don't know what's going on inside, find the source code. If it's the kernel, load up lxr ( http://lxr.linux.no/ ).

Very interesting. I spotted two minor problems with the posted code. Doing this: #define BUF_SIZE 1024 * 1024 * 5 to define a numerical constant is a bit scary, since depending on how the symbol is used it can break due to dependencies. It's better to enclose the value in parenthesis. Personally I would probably write the right hand side as (5 The first time the modification to skip entries with inode 0 is mentioned,…

On second thought (too late to edit), it's quite likely that I would not even bother with a define for this, but instead just set the size directly in the code, e.g. char buffer[5 << 20], then rely on sizeof buffer in the call where the buffer's size is needed. I prefer sizeof whenever possible.
Post reply on HN