Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Hard Drive Stats for 2018

backblaze.com

Post reply on HN
Print view View original

161–165 of 165 posts

Re: Backblaze Hard Drive Stats for 2018

#161

Post by brianwski »

Arn_Thor wrote: post #145
brianwski wrote: post #32

Earlier quoted context omitted.

Disclaimer: I work at Backblaze so I'm biased. :-) > If you have terabytes to back up, are there still any backup services left that'll let you ship them a drive for a faster initial backup? Backblaze offers a "Backblaze Rapid Ingest Fireball" to allow you to ship us 60 TBytes of data on an appliance. https://www.backblaze.com/blog/introducing-backblazes-rapid-... If you only have 2 - 10 TBytes, I suggest you get a f…

Thanks for weighing in. I wish it was that fast. I'm a new customer (formerly CrashPlan) uploading about 7TB, mostly photos and videos. Got a 500Mbps up/down line (in theory and in practice), but my daily uploading to Backblaze has never surpassed 30Mbps I've tried both low, medium and max thread numbers. In such a case, could the HDD itself be a bottleneck as it's trying to slice and dice a lot of files large and sm…

> largefiles sliced into small

Yes. Backblaze makes a copy locally of all files larger than 30 MBytes, broken into 10 MByte chunks. They’re stored on your “Temporary Scratch Disk” (which you can specify). One hint would be to put your temporary scratch disk on a fast SSD.

Personally I can get over 150 Mbits/sec upload, but I am on a PCIe SSD and have excellent latency to the datacenter. The worst case is a 5400 RPM drive powered only by USB, located in New Zealand. They would have trouble hitting even 20 Mbits/sec using the newest 6.0 client with the max of 30 threads.

Re: Backblaze Hard Drive Stats for 2018

#162

Post by Arn_Thor »

brianwski wrote: post #161
Arn_Thor wrote: post #145

Earlier quoted context omitted.

Thanks for weighing in. I wish it was that fast. I'm a new customer (formerly CrashPlan) uploading about 7TB, mostly photos and videos. Got a 500Mbps up/down line (in theory and in practice), but my daily uploading to Backblaze has never surpassed 30Mbps I've tried both low, medium and max thread numbers. In such a case, could the HDD itself be a bottleneck as it's trying to slice and dice a lot of files large and sm…

> largefiles sliced into small Yes. Backblaze makes a copy locally of all files larger than 30 MBytes, broken into 10 MByte chunks. They’re stored on your “Temporary Scratch Disk” (which you can specify). One hint would be to put your temporary scratch disk on a fast SSD. Personally I can get over 150 Mbits/sec upload, but I am on a PCIe SSD and have excellent latency to the datacenter. The worst case is a 5400 RPM d…

Because Backblaze doesn't allow program files I'm forced to upload a backup image of my C drive, and since the scratch disk needs to be bigger than the biggest file.. I don't have an SSD big enough. But not to worry, I'll be up to speed in a few weeks

Re: Backblaze Hard Drive Stats for 2018

#163

Post by brianwski »

zepearl wrote: post #132
brianwski wrote: post #129

Earlier quoted context omitted.

Correct. > if any datacenter is completely wiped out Correct. When all of our datacenters were in Sacramento, California, some customers told us they were concerned because they were ALSO in Sacramento and a meteor could wipe out both their computer, the local backup, and Backblaze's cloud backup, all in one meteor strike. While by default we put your data where it is convenient for Backblaze, we CAN work with custom…

Thanks, once more, for the nice reply :) Interesting about the new European region, for sure at least from the point of view of "locality" (I assume that from the point of view of "data ownership" the US will still consider itself "owner" of the data as the holding/legal entity (don't know what kind of company it is, but your website mentions San Mateo US) has its headquarters in the US.

> US consider itself “owner”

Well, I work most days in San Mateo, California, but 15% of the data we store for customers ALREADY comes from the EU, and more from other countries. Backblaze fully complies with all EU laws already, such as collecting VAT and passing that money through to EU countries.

Philosophically, we feel the data belongs to the customer, but we comply with all laws in that customer’s country. For the Backblaze Personal Backup product this was easiest, since it is encrypted on the customer machine before being sent. For B2 (our object storage product like Amazon S3) it got much more complicated because for the first time customers can configure it to be a publically accessible web host, so Backblaze sometimes gets served with takedown notices due to illegal content hosting.

We ABSOLUTELY comply with standard procedures the same as Amazon S3 must. Backblaze is not some crazy safe harbor for criminals hosting stolen movies. With that said, if you encrypt the data before it leaves your computer and store it in a private bucket, Backblaze has no possible way to know your file contents and we do not want to know. And we would have no way of handing that over to the US government (or the EU) even if they demanded it.

Re: Backblaze Hard Drive Stats for 2018

#164

Post by cerberusss »

StrangeDoctor wrote: post #65
fpgaminer wrote: post #54

Tangentially related. Whenever I get a new drive, I always do a "burn-in". A program writes data to the whole drive and then reads it back (reproducible random data). Is there any real justification for doing this kind of test on a new drive? Doing it takes quite awhile, so I've been wondering lately if it's even worth it. I've never found anything with it.

I do this with SD cards, I've found fakes with regards to speed and capacity. But I use ZFS for drives and just trust it to do its thing.

Yup, I do this with SD cards as well. To do this on the commandline, I just fill it up with big gigabyte-sized files. The loop uses "seq 1 16"; at least 15 files should fit on a 16 GB stick.

    $ cd /Volumes/your_new_SD_card
    $ dd if=/dev/urandom of=test.bin bs=10000000 count=1000
    $ for i in $(seq 1 16); do cp test.bin test$i.bin; done
For good measure, you could run md5/md5sum on the files.

Re: Backblaze Hard Drive Stats for 2018

#165

Post by b3lvedere »

evil-olive wrote: post #123
b3lvedere wrote: post #107

Thank you Backblaze! I love your reports. What is your procedure/policy on which disks to use in the pods? Do you try and maybe control the risk by using different harddisk brands in a single storage pod? Or do you just not care, because there have never been 3 pods dead at the same time? :) Do you still use 17 data plus 3 parity shards?

Their Q3 2018 stats had a bit of info on the lifecycle of introducing new disks: https://www.backblaze.com/blog/2018-hard-drive-failure-rates... > In Q3 we added 79 HGST 12TB drives (model: HUH721212ALN604) to the farm. While 79 may seem like an unusual number of drives to add, it represents “stage 2” of our drive testing process. Stage 1 uses 20 drives, the number of hard drives in one Backblaze Vault tome. That is,…

Thank you for the info! Much appreciated.
Post reply on HN