Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You can list a directory containing 8 million files But not with ls..

olark.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: You can list a directory containing 8 million files But not with ls..

#41

Post by sixtofour »

Ralith wrote: post #37
moe wrote: post #15

Earlier quoted context omitted.

Actually 'find' will also stat each entry no matter what. Many of the standard-tools that most people would intuitively expect to be rather optimized (find, rsync, gzip) are embarrassingly inefficient under the hood and turn belly up when confronted with data of any significant size. That probably stems from the fact that most of the development on these tools took place during a time when 1GB harddrives were "huge"…

But times have changed, and development isn't dead. Why haven't they been updated? The optimizations you imply are often straightforward and well-understood; not major undertakings to implement.

"But times have changed, and development isn't dead. Why haven't they been updated?"

Maybe because listing 8M files is not a common use case, and there just isn't the motivation to update otherwise perfectly working code. It's not an itchy problem.

Re: You can list a directory containing 8 million files But not with ls..

#42

Post by sp332 »

uxp wrote: post #20
carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

That's because linux commands are generally quiet unless something goes wrong. ./configure, make and gcc can produce pages of output even when nothing is wrong.

Re: You can list a directory containing 8 million files But not with ls..

#44

Post by archangel_one »

ck2 wrote: post #38

Why aren't files stored in subdirectories based on the first character of the filename to breakup the volume?

They were never meant to have anywhere near that number of files in the first place. The article mentions that the cleaning up of old files failed - presumably they didn't bother with subdirectories because under normal circumstances the single directory approach was working fine.

Re: You can list a directory containing 8 million files But not with ls..

#48

Post by pyre »

emmelaich wrote: post #45
machrider wrote: post #23

Earlier quoted context omitted.

Or \ls.

And you don't want to bother sorting, so: \ls -f

Or:

  env ls
Notes:

* IIRC 'env' is a built-in on csh/tcsh though, and doesn't behave like I would expect it to. You may want to read that manpage in that case.

* This is how the following works:

  #!/usr/bin/env python

Re: You can list a directory containing 8 million files But not with ls..

#49

Post by burgerbrain »

carbonica wrote: post #18

>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?

Surely it should be the other way around too. Compiling code then modifying it seems impractical. :)

Re: You can list a directory containing 8 million files But not with ls..

#50

Post by tedunangst »

JimmyMCN wrote: post #46

Wouldn't ReiserFS work better for this?

ext3 and ext4 both support b[h]tree directories, which I guess is what you're thinking of, but that would only matter for creating/deleting/looking up a particular file, not listing them. The fact the system didn't slow to a crawl creating the directory suggests that's not the problem.
Post reply on HN