Live data from Hacker News

You can list a directory containing 8 million files But not with ls..

olark.com

81–90 of 93 posts

Re: You can list a directory containing 8 million files But not with ls..

#81
post #71
post #68

Earlier quoted context omitted.

Speaking as a novice Ubuntu user, I don't even know what "install a port" means. Do you mean "open a port"?

Nope, he does not mean open a port. Ports is the name of the packaging system FreeBSD uses, and a port is the equivalent (more or less) of an RPM or deb source package. http://www.freebsd.org/ports/

Doesn't that become awfully confusing?

Re: You can list a directory containing 8 million files But not with ls..

#82
post #80
post #62

Earlier quoted context omitted.

Very interesting. I spotted two minor problems with the posted code. Doing this: #define BUF_SIZE 1024 * 1024 * 5 to define a numerical constant is a bit scary, since depending on how the symbol is used it can break due to dependencies. It's better to enclose the value in parenthesis. Personally I would probably write the right hand side as (5 The first time the modification to skip entries with inode 0 is mentioned,…

On second thought (too late to edit), it's quite likely that I would not even bother with a define for this, but instead just set the size directly in the code, e.g. char buffer[5 << 20], then rely on sizeof buffer in the call where the buffer's size is needed. I prefer sizeof whenever possible.

Using (5 And I definitely prefer defined constants for two reasons. One, it's likely you'll have to declare multiple such buffers in different places. Two, if I want to tune the parameter, I'd rather do it at the top of a source file with other such defined constants than hunting for the declaration in the code. I do agree that sizeof() is preferable when it's an option.

Re: You can list a directory containing 8 million files But not with ls..

#83
post #53

Really surprised by all the high-fiving and positive excitement going on about this article. Putting eight million files in one directory level aside, the whole basis for this event - using the filesystem as a storage layer for a k/v 'database' - is just twisted. Happy not to be working with devs like this.

Systems people tend to do the simplest, reasonable thing that works, then forget about it until it doesn't work anymore. This means that sometimes you'll make design choices that look ugly, but really, they don't matter.

See this talk by Jonathan Blow, the designer and programmer behind Braid: http://news.ycombinator.com/item?id=2689999

Re: You can list a directory containing 8 million files But not with ls..

#84
post #15

The easy way to do this is: find . -maxdepth 1 -mindepth 1 Those arguments will remove the need for find to stat each directory entry. Regardless, this is a nice walk through of low level details often overlooked.

Actually 'find' will also stat each entry no matter what. Many of the standard-tools that most people would intuitively expect to be rather optimized (find, rsync, gzip) are embarrassingly inefficient under the hood and turn belly up when confronted with data of any significant size. That probably stems from the fact that most of the development on these tools took place during a time when 1GB harddrives were "huge"…

How do you tell a file from a directory without stat()ing it? The d_type field is not portable. Since find and other tools like it need to recursively descend a directory tree, a stat() for each file to determine its type is unavoidable.

Re: You can list a directory containing 8 million files But not with ls..

#85
post #82
post #80

Earlier quoted context omitted.

On second thought (too late to edit), it's quite likely that I would not even bother with a define for this, but instead just set the size directly in the code, e.g. char buffer[5 << 20], then rely on sizeof buffer in the call where the buffer's size is needed. I prefer sizeof whenever possible.

Using (5 And I definitely prefer defined constants for two reasons. One, it's likely you'll have to declare multiple such buffers in different places. Two, if I want to tune the parameter, I'd rather do it at the top of a source file with other such defined constants than hunting for the declaration in the code. I do agree that sizeof() is preferable when it's an option.

I don't think he meant it as an optimization. It's easy to understand as 5 * 2^20, and quicker to type.

Re: You can list a directory containing 8 million files But not with ls..

#86
post #82
post #80

Earlier quoted context omitted.

On second thought (too late to edit), it's quite likely that I would not even bother with a define for this, but instead just set the size directly in the code, e.g. char buffer[5 << 20], then rely on sizeof buffer in the call where the buffer's size is needed. I prefer sizeof whenever possible.

Using (5 And I definitely prefer defined constants for two reasons. One, it's likely you'll have to declare multiple such buffers in different places. Two, if I want to tune the parameter, I'd rather do it at the top of a source file with other such defined constants than hunting for the declaration in the code. I do agree that sizeof() is preferable when it's an option.

[deleted]

Re: You can list a directory containing 8 million files But not with ls..

#87
post #42
post #20

Earlier quoted context omitted.

Throw a novice Ubuntu user into a FreeBSD system, and tell him to install a port, and 9 times out of 10 they'll freak out once they see GCC output on the screen. Nothing against Ubuntu (RedHat, SuSE, Debian, Arch, et. al.), but source compilation is something they all have been letting their users avoid for a long time. The target audience is different.

That's because linux commands are generally quiet unless something goes wrong. ./configure, make and gcc can produce pages of output even when nothing is wrong.

Yeah, but packaged source code probably shouldn't be riddled with warnings.

Re: You can list a directory containing 8 million files But not with ls..

#88
post #82

Earlier quoted context omitted.

Using (5 And I definitely prefer defined constants for two reasons. One, it's likely you'll have to declare multiple such buffers in different places. Two, if I want to tune the parameter, I'd rather do it at the top of a source file with other such defined constants than hunting for the declaration in the code. I do agree that sizeof() is preferable when it's an option.

I don't think he meant it as an optimization. It's easy to understand as 5 * 2^20, and quicker to type.

I much prefer 5 * 1024 * 1024. I have to stop and do some reasoning with 5 << 20.

Re: You can list a directory containing 8 million files But not with ls..

#89
post #81
post #71

Earlier quoted context omitted.

Nope, he does not mean open a port. Ports is the name of the packaging system FreeBSD uses, and a port is the equivalent (more or less) of an RPM or deb source package. http://www.freebsd.org/ports/

Doesn't that become awfully confusing?

Only out of context.

If your server had a network connection problem and you needed to open up a port, we're already talking about network ports, not software, so you would try to diagnose your network.

If you asked me how you could get the old game `rogue` on your system, I would tell you to go install the freebsd-games port, which has nothing to do with networking, and so it clearly means that you need to look in your ports tree.

Re: You can list a directory containing 8 million files But not with ls..

#90
post #37

Earlier quoted context omitted.

But times have changed, and development isn't dead. Why haven't they been updated? The optimizations you imply are often straightforward and well-understood; not major undertakings to implement.

"But times have changed, and development isn't dead. Why haven't they been updated?" Maybe because listing 8M files is not a common use case, and there just isn't the motivation to update otherwise perfectly working code. It's not an itchy problem.

Compressing/decompressing large quantities of data is, at the very least.
Post reply on HN