My impulse would be to modify libc's readdir() to use a larger buffer size instead of using my own C program. Would that much stupider for some reason (besides packaging/dependencies)? Do libc clients expect 32k buffers and die if they receive something else?
You can list a directory containing 8 million files But not with ls..
61–70 of 93 posts
Re: You can list a directory containing 8 million files But not with ls..
#62Excellent writeup. Computer systems are discoverable. That attitude, along with some of the basic tools (such as strace, ltrace, man pages, debuggers and a compiler) and a willingness to dive into system source code go a long way. If your tracing leads you to a library call and you don't know what's going on inside, find the source code. If it's the kernel, load up lxr ( http://lxr.linux.no/ ).
Very interesting. I spotted two minor problems with the posted code.
Doing this: #define BUF_SIZE 1024 * 1024 * 5
to define a numerical constant is a bit scary, since depending on how the symbol is used it can break due to dependencies. It's better to enclose the value in parenthesis. Personally I would probably write the right hand side as (5 The first time the modification to skip entries with inode 0 is mentioned, the code is wrong:
I did this by adding if (dp->d_ino == 0) printf(...);
This should use !=, not == (it's correct the second time, but this adds confusion).
Re: You can list a directory containing 8 million files But not with ls..
#63Wouldn't the reason be that `ls` tries to sort and stat the results?
Perhaps
ls -f1 # (one) disables sorting and just prints filenames
Re: You can list a directory containing 8 million files But not with ls..
#64Earlier quoted context omitted.
Both valid perspectives; it depends on your priorities. Some people want computers to be a black box, because when the black box works, it means less cognitive load to use.
The most interesting thing (in this subthread) is that Linux is able to be a black box. That wasn't the case not too long ago.
Well, once the hardware started working and the graphical toolkit options expanded beyond Tcl/Tk / Perl/Tk...
Ubuntu accelerated the process. Despite being a long time Linux user and programmer, I'd rather know the machine is goin' to work when I haven't done anything to mangle the beast.
I've spent hundreds, if not thousands of hours in the past just getting networking drivers to function. The brave new world of Linux is a good thing, the Interp-Only volken serve only to bolster the ecosystem, not harm it.
Re: You can list a directory containing 8 million files But not with ls..
#65>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?
Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.
I've rarely seen macports, FreeBSD ports, or NetBSD ports used as a harness to install modified versions of software. Hack jobs are almost always manual, so this purported benefit is a canard.
Re: You can list a directory containing 8 million files But not with ls..
#66Having a fan-out directory instead of putting all files at the root would have helped to avoid this problem altogether.
Here's an example. The .git/objects/ directory can grow to have up to 256 sub-directories. If an object hashes to 0a1b2c3d then it gets written to objects/0a/b2c3d. Lookups are still fast and navigating can be done without resorting to writing an 'ls' replacement.
Re: You can list a directory containing 8 million files But not with ls..
#67Earlier quoted context omitted.
Actually 'find' will also stat each entry no matter what. Many of the standard-tools that most people would intuitively expect to be rather optimized (find, rsync, gzip) are embarrassingly inefficient under the hood and turn belly up when confronted with data of any significant size. That probably stems from the fact that most of the development on these tools took place during a time when 1GB harddrives were "huge"…
The only issue I'm aware of with gzip is actually in zlib, where it stored 32-bit byte counters, but those were strictly optional and it works fine with data that overflowed them. The zlib window size may be only 32k, but bzip2 doesn't do that much better with 900k and a better algorithm, so I wouldn't consider it embarrassingly inefficient.
I was referring to the lack of SMP support in gzip (see http://www.zlib.net/pigz/).
Re: You can list a directory containing 8 million files But not with ls..
#68>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?
Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.
Speaking as a novice Ubuntu user, I don't even know what "install a port" means. Do you mean "open a port"?
Re: You can list a directory containing 8 million files But not with ls..
#69Earlier quoted context omitted.
Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.
Speaking as a novice Ubuntu user, I don't even know what "install a port" means. Do you mean "open a port"?
Ports tree is the *BSD source package manager.
Re: You can list a directory containing 8 million files But not with ls..
#70Really surprised by all the high-fiving and positive excitement going on about this article. Putting eight million files in one directory level aside, the whole basis for this event - using the filesystem as a storage layer for a k/v 'database' - is just twisted. Happy not to be working with devs like this.
The author points out it was a bug that caused this, not a valid state. I think they should probably be using sqlite anyways though.