Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can list a directory containing 8 million files But not with ls..

olark.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: You can list a directory containing 8 million files But not with ls..

#21

Post by pyre »

jesboat wrote: post #14
rarrrrrr wrote: post #10

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

H tried using find, but, in this case, the libc readdir function was the bottleneck, so find doesn't help. Often, when ls is being slow, you can speed things up drastically by disabling the things that require a stat (colors, -F, etc.) that are often added by distro's shell files (invoking ls with a full path is an easy way to disable shell aliases.) Also, sorting is on by default, which slows things for obvious reas…

  > invoking ls with a full path is an easy way to
  > disable shell aliases
Or doing this:

  > 'ls'

Re: You can list a directory containing 8 million files But not with ls..

#22

Post by mitchty »

bcx wrote: post #8
ms4720 wrote: post #4

Just curious did you try 'ls -f' to get unsorted output. I think a lot of your time may have been in getting stuff sorted for output.

I actually didn't try ls -f, but I did try find . and os.listdir and both hung with very similar straces

I hit this with a log trimming script I wrote in perl a few months back. We had one directory with about 4 million files.

Just in case you're curious I left it run and it took about 25 hours to get all of the directory entries. One with about 3 million files took 12 hours, so i'm not sure how long it would have taken if you'd have let ls run on its own accord.

Only 200k files afterwards though. :D

I think i'll poke around with this after I get a coffee, I remember stracing the interpreter and getting annoyed at its 32k or bust behavior. But since I didn't have a time limit I didn't much care about runtime. Thanks for the write up!

Re: You can list a directory containing 8 million files But not with ls..

#23

Post by machrider »

pyre wrote: post #21
jesboat wrote: post #14

Earlier quoted context omitted.

H tried using find, but, in this case, the libc readdir function was the bottleneck, so find doesn't help. Often, when ls is being slow, you can speed things up drastically by disabling the things that require a stat (colors, -F, etc.) that are often added by distro's shell files (invoking ls with a full path is an easy way to disable shell aliases.) Also, sorting is on by default, which slows things for obvious reas…

> invoking ls with a full path is an easy way to > disable shell aliases Or doing this: > 'ls'

Or \ls.

Re: You can list a directory containing 8 million files But not with ls..

#24

Post by sofal »

bcx wrote: post #9
bpierre wrote: post #6

Earlier quoted context omitted.

Better cached version: http://webcache.googleusercontent.com/search?q=cache:KBsyzf3...

no need for cache anymore. Word to the wise, make sure supercache is actually caching when copying wordpress installs :-)

The cache is the only way it will display properly on my Android phone.

Re: You can list a directory containing 8 million files But not with ls..

#25

Post by blhack »

carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Absolutely. The sad reality is that, to a lot of linux users right now (and I'm talking about people that know they're running linux, not android users), it's a black box.

Software is "packages" that you install with "synaptic".

Source code? Compiler? What's that?

Re: You can list a directory containing 8 million files But not with ls..

#29

Post by carbonica »

blhack wrote: post #25
carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Absolutely. The sad reality is that, to a lot of linux users right now (and I'm talking about people that know they're running linux, not android users), it's a black box. Software is "packages" that you install with "synaptic". Source code? Compiler? What's that?

This seemed like a developer-targeted article, though. I mean, who is having trouble with even 1 million directory entries, let alone 8 million, who is afraid of a compiler?

Re: You can list a directory containing 8 million files But not with ls..

#30

Post by bcx »

carbonica wrote: post #29
blhack wrote: post #25

Earlier quoted context omitted.

Absolutely. The sad reality is that, to a lot of linux users right now (and I'm talking about people that know they're running linux, not android users), it's a black box. Software is "packages" that you install with "synaptic". Source code? Compiler? What's that?

This seemed like a developer-targeted article, though. I mean, who is having trouble with even 1 million directory entries, let alone 8 million, who is afraid of a compiler?

Developers who only write in interpreted languages :-)
Post reply on HN