Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My experience with using cp to copy 432 million files (39 TB)

lists.gnu.org

Post reply on HN
Print view View original

261–267 of 267 posts

Re: My experience with using cp to copy 432 million files (39 TB)

#261

Post by pedrocr »

pflanze wrote: post #259
pedrocr wrote: post #258

Earlier quoted context omitted.

> My algorithm works with the nlink==1 optimization just as well: just don't put those files into the file list that needs to be sorted, instead copy them immediately or put them into a list that doesn't need sorting. If you do that then your extra storage requirements are now O(paths to inodes with nlinks > 1) which is still strictly higher than O(inodes with nlinks>1). You're also doing two passes, one to enumerate…

> O(paths to inodes with nlinks > 1) .. O(inodes with nlinks>1) Yes, that's right, I ignored that part in the calculation above. By using the filename+parent pointer approach for path compression, that could be improved. (The current version of megacp also stores all necessary stat information (size, uid, gid, perms) in the tempfile to avoid a second stat call; this could be dropped if it makes matters worse.) Also,…

> Also, quite likely using a fast compression algorithm like lzop on the file would make disk access faster and disk storage requirements lower.

>Using an online algorithm is the whole point, I've been saying that from the start. And I remember that Knuth wrote about them so I conclude that they exist for sort, and I quite expect that GNU sort(1) is already sorting big files efficiently (limited testing seems to confirm that, files several times the size of RAM sort just fine in the low single digit minute count).

The cp algorithm already solves this just fine and my proposed reimplementation on-disk solves the RAM usage problem. You're trying to make a strictly worse algorithm work. It both requires more storage and is heavier than it needs to be (full sort vs just a map). What do you expect to gain from it?

Re: My experience with using cp to copy 432 million files (39 TB)

#262

Post by [deleted] »

pedrocr wrote: post #261
pflanze wrote: post #259

Earlier quoted context omitted.

> O(paths to inodes with nlinks > 1) .. O(inodes with nlinks>1) Yes, that's right, I ignored that part in the calculation above. By using the filename+parent pointer approach for path compression, that could be improved. (The current version of megacp also stores all necessary stat information (size, uid, gid, perms) in the tempfile to avoid a second stat call; this could be dropped if it makes matters worse.) Also,…

> Also, quite likely using a fast compression algorithm like lzop on the file would make disk access faster and disk storage requirements lower. >Using an online algorithm is the whole point, I've been saying that from the start. And I remember that Knuth wrote about them so I conclude that they exist for sort, and I quite expect that GNU sort(1) is already sorting big files efficiently (limited testing seems to conf…

[deleted]

Re: My experience with using cp to copy 432 million files (39 TB)

#263

Post by halfcat »

coned88 wrote: post #256
halfcat wrote: post #244

Earlier quoted context omitted.

10^14 is bits. When you divide 10^14 by 8 you get 12.5 trillion bytes, or 12.5 TB. If you have 4x 4TB drives in RAID5, and one fails, then in order to rebuild with a replacement drive, you have to read all data from all surviving drives (3 x 4TB = 12TB). Here is an example from a manufacturer: http://www.seagate.com/files/www-content/product-content/con... They call it "Non-recoverable Read Errors per Bits Read" and…

OK i see now but if it needs to read 12TB from all drives together then that's still far from 12.5TB per drive which is the limit. That's where I am confused.

Say we have a department of a company with 36 employees, and one pair of dice. We decide that if any person out of the entire department rolls a 12, then everyone in the department will be fired. The chance of rolling a 12 is 1/36. It doesn't matter if one person keeps rolling the dice, or if they take turns, the chances of everyone being fired are close to 100%.

The same is true for a disk array. Each read operation is an independent event (for the purpose of doing this math). The chance of one URE happening is 1/(10^14) for every bit read. It doesn't matter which disk it happens on. When it happens, the entire array is failed.

Also 12.5 TB is not a hard limit, just an average. The URE could happen on the very first read operation, or you might read 100 TB without a URE.

Re: My experience with using cp to copy 432 million files (39 TB)

#264

Post by davidu »

lysium wrote: post #260
davidu wrote: post #211

Earlier quoted context omitted.

http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...

Thanks for the link! The article says, that due to the read error rate and the size of today's disks, RAID 5 and RAID 6 have (kind of) lost their purpose.

Yep, mathematically no longer safe.

Re: My experience with using cp to copy 432 million files (39 TB)

#265

Post by scott_karana »

novaleaf wrote: post #199
fintler wrote: post #93

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems. https://github.com/hpc/dcp We got an IEEE paper out of it: http://conferences.computer.org/sc/2012/papers/1000a015.pdf A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/h…

man! and here I am feeling like a champ conquering NTFS's long file name limitations :/

Is your conquering public? We have issues occasionally, and a toolkit would be nice :)

Re: My experience with using cp to copy 432 million files (39 TB)

#266

Post by otterley »

dbbolton wrote: post #13

>We use XFS Why?

ext2/3/4 have terrible large-directory performance. If you create 100,000 files in a directory in an ext2/3/4 filesystem, enumerating said files (ls, readdir, etc.) becomes unbearably slow -- and the slowness persists even after you delete all but one of the files in the directory.

Re: My experience with using cp to copy 432 million files (39 TB)

#267

Post by 72deluxe »

peterwwillis wrote: post #200
72deluxe wrote: post #165

Earlier quoted context omitted.

It's also an astute observation that is incredibly true! Most sites now incorporate JavaScript. I remember when it was "new-fangled" (and DHTML anyone?) and everyone put those annoying cursor trackers that trailed blobs from where your cursor was.

Uh oh, is it nostalgia time already? What I mainly remember is two things: suddenly everyone had to do client-side form checking to prevent people from typing dashes in a phone number field, and everyone tried to imitate Java without actually using Java. It seems like any way you could enhance a page by combining mouseover events, gifs and loading content dynamically you would throw into a page, even if it made the u…

I have those memories too, on Windows 95 or 98? I remember IE4 being released and it could do a lot more than IE3, plus it was more "forgiving" with sloppy HTML compared to Netscape.

I remember those myriads of mouseover buttons.

Post reply on HN