The author has a great tip for kernel/filesystem developers:
"Perhaps the buffer should be dynamically set based on the size of the directory entry file"
This would eliminate the readdir() bottleneck.
You can list a directory containing 8 million files But not with ls..
51–60 of 93 posts
Re: You can list a directory containing 8 million files But not with ls..
#52Excellent case for not giving root.
What does root have to do with this?
Article illustrates numerous manner of ways of doing things incorrectly.
The premise of the article is a bad precedent for stable environments: let's bend the OS so that it plays nicely with what's clearly misuse and misunderstanding of filesystems.
The only way eight million files should ever end up in a single directory level is by accident, and that's not the case in the scenario outlined here.
Re: You can list a directory containing 8 million files But not with ls..
#53Really surprised by all the high-fiving and positive excitement going on about this article.
Putting eight million files in one directory level aside, the whole basis for this event - using the filesystem as a storage layer for a k/v 'database' - is just twisted.
Happy not to be working with devs like this.
Re: You can list a directory containing 8 million files But not with ls..
#54Earlier quoted context omitted.
What does root have to do with this?
Article illustrates numerous manner of ways of doing things incorrectly. The premise of the article is a bad precedent for stable environments: let's bend the OS so that it plays nicely with what's clearly misuse and misunderstanding of filesystems. The only way eight million files should ever end up in a single directory level is by accident, and that's not the case in the scenario outlined here.
Typically there are on the order of 2000 files in a directory. It was a mistake.
Re: You can list a directory containing 8 million files But not with ls..
#55I feel compelled to point out that perl can do this rather easily. But I really did like a little more low level on how a directory is listed.
http://blogs.perl.org/users/randal_l_schwartz/2011/03/perl-t...
Re: You can list a directory containing 8 million files But not with ls..
#56Really surprised by all the high-fiving and positive excitement going on about this article. Putting eight million files in one directory level aside, the whole basis for this event - using the filesystem as a storage layer for a k/v 'database' - is just twisted. Happy not to be working with devs like this.
Indeed,
Sure, you can find out a lot by doing things you really shouldn't do, like using the directory system as k/v store.
But in the end, you should still learn the lesson that it is really a bad idea.
Re: You can list a directory containing 8 million files But not with ls..
#57I suspect the author is incorrect in his claim that reading in 32k chunks is responsible for the slowness. Due to read ahead and buffering, Unix-like systems tend to do reasonably well on small reads. Yes, big reads are better, but small reads are not unreasonably slow. To test this, he should try "ls | cat". On large directories that often runs many orders of magnitude faster than "ls". This is because, I believe, l…
> To test this, he should try "ls | cat".
Running /bin/ls will bypass the alias.
Re: You can list a directory containing 8 million files But not with ls..
#58The author has a great tip for kernel/filesystem developers: "Perhaps the buffer should be dynamically set based on the size of the directory entry file" This would eliminate the readdir() bottleneck.
readdir() is implemented in libc, not the kernel. The kernel interface is getdents() and that has a configurable buffer size.
Re: You can list a directory containing 8 million files But not with ls..
#59I suspect the author is incorrect in his claim that reading in 32k chunks is responsible for the slowness. Due to read ahead and buffering, Unix-like systems tend to do reasonably well on small reads. Yes, big reads are better, but small reads are not unreasonably slow. To test this, he should try "ls | cat". On large directories that often runs many orders of magnitude faster than "ls". This is because, I believe, l…
The original onus for the post was python's os.listdir() which as far as I know doesn't stat(). ls, just made the blog post more interesting :-). I was surprised that the 32K reads were taking so long. It's possible since it was on a virtualized disk ("in the cloud") that something else was slowing down disk IO (like Xen). But I can assure you that a larger read buffer performed much better in this given scenario. I'…
This is just a hypothesis based on very little actual knowledge, but perhaps a very long scheduling interval is responsible for the slowness with smaller reads? Consider this scenario: the virtualization hypervisor is on a reasonably loaded system, and decides to block the virtual machine for every single read. Since the physical system has several other VMs on it, whenever the VM in question loses its time slice it has to wait a long time to get another one. Thus, even if the 32K read itself happens quickly, the act of reading alone causes a delay of n milliseconds. If you increase the read size, your VM still gets scheduled 1000/n times per second, but each time it gets scheduled it reads 5MB instead of 32K.
Re: You can list a directory containing 8 million files But not with ls..
#60>>> "Don’t be afraid to compile code and modify it" I was a bit thrown by this advice. Are there folks out there that are afraid to compile code and modify it?
Are there folks out there that are afraid to compile code and modify it?
Some developers (myself included) may have a general preference for sticking with the "official" packages to avoid extra work when bringing up a new machine or migrating to a new distro version.