Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Storage Pod 5.0

backblaze.com

Post reply on HN
Print view View original

211–220 of 222 posts

Re: Backblaze Storage Pod 5.0

#211

Post by sofaofthedamned »

nickpsecurity wrote: post #184
jamwt wrote: post #173

Earlier quoted context omitted.

Wow, very cool! So, candidate for offloading that's really commonly considered is "scrubbing". Essentially, protecting against passive bit flips by checking your data against your own checksums. If you have some N TB of data under management, it's really expensive to be striping these large serial reads all the way to an application's userland. So finding ways to keep these as low-priority "background" reads (that al…

"So, candidate for offloading that's really commonly considered is "scrubbing". Essentially, protecting against passive bit flips by checking your data against your own checksums." "Or even offload it to hardware--some custom host adapter, possibly, using custom ATA/SCSI commands to control it and query it. (and the `sg` driver)." Now you're thinking. Chips like Octeon already do hardware acceleration of checksums an…

If you've got custom drives with custom firmware, why does the scrub need to leave the drive? Couldn't they have a command that kicks off the scrub in-drive, with only a list of failed blocks returned?

Re: Backblaze Storage Pod 5.0

#212

Post by nickpsecurity »

sofaofthedamned wrote: post #211
nickpsecurity wrote: post #184

Earlier quoted context omitted.

"So, candidate for offloading that's really commonly considered is "scrubbing". Essentially, protecting against passive bit flips by checking your data against your own checksums." "Or even offload it to hardware--some custom host adapter, possibly, using custom ATA/SCSI commands to control it and query it. (and the `sg` driver)." Now you're thinking. Chips like Octeon already do hardware acceleration of checksums an…

If you've got custom drives with custom firmware, why does the scrub need to leave the drive? Couldn't they have a command that kicks off the scrub in-drive, with only a list of failed blocks returned?

Are we talking about firmware as drivers executing on host or the microcontrollers on the drives? My proposal works on adapter in between the two or could replace HD microcontroller if you had willing vendor. Doing it on host sucks host CPU requiring more powerful and energy consuming processors given No of drives and data amount.

Idea is to move straight-forward, possibly parallelizable algorithms off general-purpose, legacy CPU's onto cheaper, efficient ones more suited to algorithm. It wasn't uncommon for FPGA's to get an eightfold boost in performance on stuff like this in various experiments.

In my field, high assurance INFOSEC, the main advantage is the use in inline, media encryptors. I really just knocked off specs and ideas of NSA's Type 1 IME given it was the best:

http://m.nsa.gov/ia/programs/inline_media_encryptor/

Also, Orange Book required full mediation and even SCOMP system mediated hardware I/O. Copying these lessons from the past made my design immune to DMA, firmware, and SMM attacks that were later discovered. Like above, offloading gets performance boost cuz inline I/O handles interrupt handling, prevents many cache flushes, and supports hardware accelerators. Also, keygen and storage stay off main server. Fun times.

Re: Backblaze Storage Pod 5.0

#213

Post by sharmajai »

atYevP wrote: post #160
mrmondo wrote: post #136

Great work Backblaze! I really appreciate the open, knowledge sharing nature of your hardware designs. This is one of the main reasons I am a Backblaze customer, I must say however that the lack of open source and headless Linux client software is seriously pushing me away, it's something I think many people have been waiting for years for, especially given the product licensing model is designed as such to backup a…

Yev from Backblaze -> We're working on it, in a fashion. With Backblaze B2 you'll be able to push from a Linux machine using CLIs/APIs and get those computers backed up as well! Slightly different, but it's the best way for us to help Linux users!

It is probably unfair to ask, but is there a way to get an early invite for B2 (I already signed up but can't contain my excitement; email: HN username at gmail)?. Also, have there been any comparisons of B2 done with Amazon Cloud Drive's unlimited everything plan for $59 a year (with break even when compared to B2 at around a TB)? They do provide an API, although I have yet to try it out.

Re: Backblaze Storage Pod 5.0

#214

Post by greedo »

budmang wrote: post #46
toomuchtodo wrote: post #23

Earlier quoted context omitted.

I use a Backblaze box as an Archive Team staging environment! Thank you!

Very cool. I remember an article I read about a veteran music journalist who applied for a job and was told he had no experience. Turned out every magazine he worked for over 10+ years had shut down and there was not a single online reference of his articles. Love that you're fighting the good fight (along with the Internet Archive) to keep a history of the Internet.

From 2000 through 2006 I freelanced for a middle tier publisher of computer magazines. Wrote 4-5 articles a month, probably 1/2 million words. The publisher had a paywall for their online versions of the print magazines, but really didn't put much effort into promoting it since print had been so lucrative. Over the last three years they shuttered the majority of the print magazines (and their corresponding online editions), and now it's all gone. The wayback machine/Internet Archive only has a limited number of pages available. Sad to realize that a lot of the work we do (whether it's writing or creating systems) is ephemeral...

Re: Backblaze Storage Pod 5.0

#215

Post by coldtea »

atYevP wrote: post #180
coldtea wrote: post #177

Earlier quoted context omitted.

Looks nice! Does it got any kind of integration with stuff you've uploaded through the regular Backblaze app? (e.g. moving stuff over to B2 once it has been uploaded as a regular BB backup).

We're treating them like two different products, so no integration yet, but if we see that a sizeable of folks would want to do both or transfer their data over, we might go down that path.

I think it would solve the "user-accessible UI" for making the initial move of the data to storage.

I can write some script myself to move over my stuff directly, but even as a programmer I'd rather not :-)

I think you're plan for it is that third parties might take that task. So, maybe Arq will.

https://twitter.com/arqbackup/status/650634416067342336

Re: Backblaze Storage Pod 5.0

#216

Post by atYevP »

coldtea wrote: post #215
atYevP wrote: post #180

Earlier quoted context omitted.

We're treating them like two different products, so no integration yet, but if we see that a sizeable of folks would want to do both or transfer their data over, we might go down that path.

I think it would solve the "user-accessible UI" for making the initial move of the data to storage. I can write some script myself to move over my stuff directly, but even as a programmer I'd rather not :-) I think you're plan for it is that third parties might take that task. So, maybe Arq will. https://twitter.com/arqbackup/status/650634416067342336

Arq says they have no plans to do it in the near term at the moment, but hopefully once we open it to the public they'll want to integrate. Worst case, if we make a bajillion dollars, we'll hire some more devs to whip something together :P

Re: Backblaze Storage Pod 5.0

#217

Post by Twirrim »

PhantomGremlin wrote: post #175
benjaminl wrote: post #134

Earlier quoted context omitted.

One thing to keep in mind is that Backblaze keeps 1.15 copies of your data, while Amazon S3 and the like keep multiple full geographically redundant copies of your data. For example Microsoft Azure Cloud Storage keeps 3 to 6 copies of your data, depending on the tier, [1] when you compare that to the 1.15 copies in Backblaze B2 you start to understand where the pricing difference comes from. For most use cases this d…

full geographically redundant copies of your data In the case of Amazon, I wouldn't be so certain that they're geographically redundant copies. I raised this issue on HN a few months ago and someone replied that within an availability zone the data is "a bunch of (datacenters) that are close to each other". https://news.ycombinator.com/item?id=10231677 I like how for GRS Microsoft specifically says "a second datacent…

Around about here in the talk: https://youtu.be/JIQETrFC_SQ?t=14m38s James Hamilton talks precisely what the definitions of AZs and regions are :)

Re: Backblaze Storage Pod 5.0

#218

Post by eiopa »

bartvk wrote: post #208
eiopa wrote: post #111

I really love these, and I desperately wanted to be a Backblaze customer, but I was put off by the abysmal desktop software :( Storage execution seems top-notch. End-user software deserves some love.

Abysmal compared to what? I have experience with the Mac client, and it's rock solid, and very well executed.

The Windows client is bad. Can't speak about the osx one.

Re: Backblaze Storage Pod 5.0

#219

Post by beagle3 »

garblegarble wrote: post #131
codemac wrote: post #83

Earlier quoted context omitted.

> you can get a lot bigger than 6T Come on - most companies reasonably can't. There's a WD 10T smr/helium drive, and seagate is selling that crap 8T video drive. Unless you're talking ssd 3d/TLC density (more $), or the vapor-ware 20T drives that some storage vendors hint at and don't let me file a PO for, I literally don't know what you're referring to (now I'm getting where I can't name names) The reason I'm compla…

>seagate is selling that crap 8T video drive Slightly off-topic but could you go into some detail on what you think the drawbacks are for the Seagate 8TB SMR drive? I've been thinking about getting a few for large files while keeping them online-accessible, I've found it difficult to get the thoughts of grizzled IT folks on the matter :-)

I would like that info as well.

I couldn't find any reliable data, but I did find a lot of reviews that said the 8TB drives tend to die within a few months; I'm holding on to the 6TBs for now.

Re: Backblaze Storage Pod 5.0

#220

Post by beagle3 »

atYevP wrote: post #183
unixhero wrote: post #172

Why are they not offering 8GB drives. It's late 2015.

The most important thing for us is Cost per GB. Right now the 4TB drives we're using are still much cheaper than the 8TBs, so we go with the 4s, but the Pods work with 8TBs if you want to drop the dough on them!

I buy ~20 drives/month (a drop in the bucket compared to you guys), but I find 6TB drives are fast, reliable, and cost the same per TB as the 4TB drives (so, taking the NAS/Pod cost into account, cheaper per online TB).

Are 4TB drives significantly cheaper for you? Or is there some other constraint at play against the 6TB?

Post reply on HN