Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

135TB for $7,384 - Backblaze Pod 2.0

zdnet.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: 135TB for $7,384 - Backblaze Pod 2.0

#41

Post by wmf »

whakojacko wrote: post #12
brianwski wrote: post #9

Earlier quoted context omitted.

At the lowest level there are three RAID groups in each pod. Each RAID group is made of 15 drives configured in software RAID 6 with 2 parity drives. This means you can lose 2 dives and the data is entirely safe and intact. If 3 or more drives completely fail simultaneously (not just pop out of the RAID group or power down, but where that drive is lost forever, like it will never power up again) you will lose at leas…

While you are here....Why only a single boot drive? It seems like an obvious failure point, and at only $40 per drive I would think soft raid 1 would be no-brainer for reliability. Regardless though, super impressed by the work into rolling your own hardware, hope you guys continue to do well.

Personally, I'd eliminate the boot drive and PXE boot.

Re: 135TB for $7,384 - Backblaze Pod 2.0

#42

Post by brianwski »

barrkel wrote: post #36
brianwski wrote: post #14

Earlier quoted context omitted.

We would like to, we just haven't had time to get it done yet. It runs internally, but is lacking an installer and a GUI, and we would need to prioritize and choose one or more Linux distributions to launch with. Ubuntu is an obvious choice (we focus more on desktop backup than on servers). But some people also ask for CentOS and a few others. It bums me out the Linux community has not solved binary compatibility any…

I like the way CrashPlan works. I use it to back up both Windows and Solaris. The GUI is Java, but I only run it on Windows; the GUI communicates over TCP to the Solaris machine. This seems to me the best compromise. Similarly, the CrashPlan service itself is written in Java and for binary compatibility, only relies on a little binary library loaded via JNI to accommodate the differences between Linux and Solaris, no…

(Backblaze founder speaking here) - I have always liked the CrashPlan company and they seem like a smart focused team with a solid product. And I've heard other reports that CrashPlan works well on Linux, so people should definitely check them out and give them a fair try.

Re: 135TB for $7,384 - Backblaze Pod 2.0

#43

Post by silvestrov »

brianwski wrote: post #34
db48x wrote: post #30

Earlier quoted context omitted.

Merkle Trees FTW, but it sounds like you've basically reimplemented ZFS. Have you tried using ZFS on one of these?

Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris. ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet. By the way, at Backblaze I've felt like we have had to implement several things that I wou…

The OSX Lion download is +3 GB, so Apple customers must take care of TCP errors (or at least Apple Support would be swamped with support requests).

Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correction.

Re: 135TB for $7,384 - Backblaze Pod 2.0

#44

Post by brianwski »

wmf wrote: post #41
whakojacko wrote: post #12

Earlier quoted context omitted.

While you are here....Why only a single boot drive? It seems like an obvious failure point, and at only $40 per drive I would think soft raid 1 would be no-brainer for reliability. Regardless though, super impressed by the work into rolling your own hardware, hope you guys continue to do well.

Personally, I'd eliminate the boot drive and PXE boot.

We're considering a PXE boot solution (among other solutions) just to keep all 200 pods (and growing!) updated to the latest Debian. But we also use the boot drive for a few other things like error logging and such. But the idea of eliminating the boot drive entirely is not a bad idea, we could drop the logs in a folder at the top of the data drives. We already (selectively) mirror various excerpts of the logs to other machines in case the whole pod disappears from the network so we have some history and understanding of what was going on when it went missing.

Re: 135TB for $7,384 - Backblaze Pod 2.0

#45

Post by brianwski »

silvestrov wrote: post #43
brianwski wrote: post #34

Earlier quoted context omitted.

Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris. ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet. By the way, at Backblaze I've felt like we have had to implement several things that I wou…

The OSX Lion download is +3 GB, so Apple customers must take care of TCP errors (or at least Apple Support would be swamped with support requests). Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correc…

Yep, Apple solves this problem by using their "AppStore" application to reliably download the file in chunks (like you suggested). My complaint is that Apple should extend Safari to be able to download the OS X Lion download reliably. (Then let the rest of us use that protocol.) It seems a bit insane that the OS group at Apple had to write their own custom download application, but did not think to extend their web browser to allow for "restartable downloads that are no longer corrupted".

Microsoft does the same thing -> they have a custom application to download their OS updates reliably. They do not use Internet Explorer's regular download capability because of the limitations.

It baffles me that the big browser companies (Google with Chrome, Apple with Safari, Microsoft with IE) are not interested in building a standard reliable downloader. I mean, what else does a web browser really do?

Re: 135TB for $7,384 - Backblaze Pod 2.0

#46

Post by raphman »

brianwski wrote: post #35
nantes wrote: post #26

Earlier quoted context omitted.

Any chance of releasing the XML file specs so someone in the community can write an OSS GUI?

See my other post on this. It is extremely simple XML anybody would understand instantly. In fact, some people have already scripted changes to the XML files running on Mac to increase the "throttle" at night and decrease the "throttle" during the day. We'll be giving the Linux client some more thought over the next few days.

Thanks!

Re: 135TB for $7,384 - Backblaze Pod 2.0

#47

Post by dmpatierno »

brianwski wrote: post #35
nantes wrote: post #26

Earlier quoted context omitted.

Any chance of releasing the XML file specs so someone in the community can write an OSS GUI?

See my other post on this. It is extremely simple XML anybody would understand instantly. In fact, some people have already scripted changes to the XML files running on Mac to increase the "throttle" at night and decrease the "throttle" during the day. We'll be giving the Linux client some more thought over the next few days.

For anyone wondering, I just did this on OS X to adjust the throttle overnight:

  > crontab -l
  0  1 * * * sed -E -i .bak 's/net_throttle="[0-9]+"/net_throttle="100"/' /Library/Backblaze/bzdata/bzinfo.xml
  0 10 * * * sed -E -i .bak 's/net_throttle="[0-9]+"/net_throttle="40"/'  /Library/Backblaze/bzdata/bzinfo.xml

Re: 135TB for $7,384 - Backblaze Pod 2.0

#48

Post by SoftwareMaven »

mrb wrote: post #2

Direct link to Backblaze's blog post: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... They got rid of the PCI bottleneck by switching to PCIe, a bottleneck which surprised me when they designed version 1.0 of their pod. They could have gone PCIe at the time, I maintain http://blog.zorinaq.com/?e=10 and they were SATA controllers at the time that met their technical requirements (nr. of ports, Linux su…

According to the post above, they were using three PCIe and one PCI.

"In the first generation storage pod, we ran out of the faster PCIe slots and had to use one slower PCI slot, creating a bottleneck."

Re: 135TB for $7,384 - Backblaze Pod 2.0

#49

Post by ghoul2 »

The blog post provides some fascinating data, thanks Backblaze!

$2100 per month for an entire rack worth of Pods (space, power, connectivity) $74,000 is the cost to build 10 Pods to fill that rack. If the Pods are assumed to have a lifetime of 3 years (most will last longer, but lets depreciate at this rate), and if the cost of capital is 20%/year, this equates to a monthly "payment/amortization" cost of $2750. Thus leading to a total cost per rack of $4850 ~ lets say $5000.

1350TB of raw storage is provided by this rack, which can be scaled to 13/15 to account for RAID6 (as revealed by brianwski here) - thus leaving 1170TB available for use. FS overhead etc, lets take this to provide 1PB of storage.

So, essentially, storage costs backblaze about 0.5 cents ($0.005) /GB-month. There are other costs ofcourse, Sean (amongst others) needs to get paid, etc. To go by a common thumb rule, for a minimum sale price, one third of sale price should be profit, another third should be org expenses/marketing/everything else and the remaining third should be the actual cash cost of the building/providing the product/service.

So very roughly, Backblaze could provide storage at about 1.5 cents /GB-month. Factor in 3-way software-level redundancy of data, and you are now upto 5 cents/GB-months for a very high quality storage service.

Contrasting this with 15c/GB-month that Amazon charges (in addition to transfer charges), I do have to wonder why Backblaze wants to stick to the "unlimited desktop backups" business. Even Google storage charges 17c/GB-month, in addition to per-request charges.

Its quite possible I have some factors wrong here and If anyone can spot anything wrong, I'd like to know. Nevertheless, it seems from the numbers provided that backblaze could make a killing in this market. I know I would be interested in using a pure storage backend - equivalent to S3. I use tarsnap and if tarsnap could reduce its backend costs by two-thirds, I know I'd be very happy.

What am I missing?

edit: wrote PB instead of TB. Numbers remain correct, though

Re: 135TB for $7,384 - Backblaze Pod 2.0

#50

Post by thaumaturgy »

brianwski wrote: post #23
whakojacko wrote: post #12

Earlier quoted context omitted.

While you are here....Why only a single boot drive? It seems like an obvious failure point, and at only $40 per drive I would think soft raid 1 would be no-brainer for reliability. Regardless though, super impressed by the work into rolling your own hardware, hope you guys continue to do well.

I just asked, and found out we actually HAVE had a number of boot drives fail in our fleet of 200 pods. Most decisions in the pod are around saving money, so our initial thoughts were just that no customer data is on the boot drive so it isn't all that important. But don't get me wrong, there are SO MANY GOOD opportunities to improve the pod, Backblaze just stops working on the pod when it does what is needed for us…

(Just FYI, I'm sure you guys have already thought of or experimented with this...)

We've had good luck so far with using small USB flash drives for booting big file servers. We keep the drive image pretty generic and if there's a problem with one, we just replace it with a cloned USB flash drive and reboot, no problem.

It doesn't seem to hurt performance at all for these kinds of uses -- although we do set it up without swap to keep the life of the USB flash device reasonable, which might or might not work in your case.

Post reply on HN