You can list a directory containing 8 million files But not with ls..

#1

Post by bcx » Mon, Aug 15, 2011, 9:34 PM UTC

You can list a directory containing 8 million files But not with ls..
olark.com

Re: You can list a directory containing 8 million files But not with ls..

#2

Post by scott_s » Mon, Aug 15, 2011, 9:51 PM UTC

Excellent writeup. Computer systems are discoverable. That attitude, along with some of the basic tools (such as strace, ltrace, man pages, debuggers and a compiler) and a willingness to dive into system source code go a long way. If your tracing leads you to a library call and you don't know what's going on inside, find the source code. If it's the kernel, load up lxr (http://lxr.linux.no/).

Re: You can list a directory containing 8 million files But not with ls..

#3

Post by nbashaw » Mon, Aug 15, 2011, 9:58 PM UTC

Sorry our blog is down now - we're trying to get it back up ASAP!

Re: You can list a directory containing 8 million files But not with ls..

#4

Post by ms4720 » Mon, Aug 15, 2011, 10:01 PM UTC

Just curious did you try 'ls -f' to get unsorted output. I think a lot of your time may have been in getting stuff sorted for output.

Re: You can list a directory containing 8 million files But not with ls..

#5

Post by mjpizz » Mon, Aug 15, 2011, 10:07 PM UTC

nbashaw wrote: post #3

Sorry our blog is down now - we're trying to get it back up ASAP!

looks like it's fine now, but worst case you can access the cached version:

http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

Re: You can list a directory containing 8 million files But not with ls..

#6

Post by bpierre » Mon, Aug 15, 2011, 10:13 PM UTC

mjpizz wrote: post #5

nbashaw wrote: post #3

Sorry our blog is down now - we're trying to get it back up ASAP!

looks like it's fine now, but worst case you can access the cached version: http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

Better cached version: http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

Re: You can list a directory containing 8 million files But not with ls..

#7

Post by bcx » Mon, Aug 15, 2011, 10:15 PM UTC

Should be fixed completely and fast now, SPW was a new blog so the cache config was support wrong :-)

Re: You can list a directory containing 8 million files But not with ls..

#8

Post by bcx » Mon, Aug 15, 2011, 10:17 PM UTC

ms4720 wrote: post #4

Just curious did you try 'ls -f' to get unsorted output. I think a lot of your time may have been in getting stuff sorted for output.

I actually didn't try ls -f, but I did try find . and os.listdir and both hung with very similar straces

Re: You can list a directory containing 8 million files But not with ls..

#9

Post by bcx » Mon, Aug 15, 2011, 10:18 PM UTC

bpierre wrote: post #6

mjpizz wrote: post #5

Earlier quoted context omitted.

looks like it's fine now, but worst case you can access the cached version: http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

Better cached version: http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

no need for cache anymore. Word to the wise, make sure supercache is actually caching when copying wordpress installs :-)

Re: You can list a directory containing 8 million files But not with ls..

#10

Post by rarrrrrr » Mon, Aug 15, 2011, 10:22 PM UTC

The easy way to do this is:

  find . -maxdepth 1 -mindepth 1

Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.