Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze B2 Cloud Storage

backblaze.com

Post reply on HN
Print view View original

181–190 of 237 posts

Re: Backblaze B2 Cloud Storage

#181

Post by jbeda »

devit wrote: post #171
jbeda wrote: post #147

Earlier quoted context omitted.

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

No, it's pretty much totally unnecessary. The API works on top of TLS, which already includes cryptographic authentication of all data (usually via SHA-1/2 HMAC or AES-GCM). The hash would be computed at the client right after reading from disk and right before TLS enryption, and since they seem to terminate TLS at the storage server it would be computed right after TLS decryption and right before storage, so it does…

When operating at scale, you will, once in a while, have corruption. Even if you use ECC RAM, once in a while you'll have a double bit flip. And it doesn't look like Backblaze uses ECC (https://www.backblaze.com/blog/storage-pod-4-5-tweaking-a-pr...) despite good evidence that ECC is necessary (PDF: http://static.googleusercontent.com/media/research.google.co...). Even if you do have ECC, you'll once in a while have a bad NIC that with HW offload that will corrupt the TCP stream silently.

This is all rare, but it does happen. This is why the GCS team wants to know if you are seeing corruption on file upload as it might be some bad hardware failing in a non-obvious way.

Re: Backblaze B2 Cloud Storage

#182

Post by scosman »

brianwski wrote: post #69
devit wrote: post #56

Finally some more reasonable prices in this space. Eventually it could make sense for Backblaze to partner with someone like DigitalOcean or Linode and offer low cost bulk storage and low cost virtualization colocated in the same datacenter: these services seem to be a perfect complement for each other.

Brian from Backblaze here. Yeah, our B2 storage may not be a good solution for an application that has to do a lot of analysis on the data over and over again. In Amazon S3 you don't pay for transfers between EC2 and S3, so computing on your data is only as expensive as buying the EC2 time. Since Backblaze doesn't yet offer the EC2 functionality you would need to download your data to analyze it. What I'd really like…

Amazon offers interconnects with reduced transfer costs: https://aws.amazon.com/directconnect/

A real deal breaker is if you need to use an EC2 server to proxy the upload for any reason (content validation). The transfer into EC2 is free, but it's 9 cents for each GB out (18 months of storage cost).

@brianwski - any suggestions here?

Re: Backblaze B2 Cloud Storage

#183

Post by dfinlay »

brianwski wrote: post #106
aidenn0 wrote: post #85

Reading the API, it seems that I need to precalculate a SHA-1 before uploading? This makes it impossible to stream data to b2 from another source, I'll need to store it first then send to b2.

Right now your only option would be to "buffer" packets of say 1 MByte in RAM, calculate the SHA-1, then store them as separate files in Backblaze B2. We do plan to add file offset access and larger file support very soon, so you would be able to append a 1 MByte chunk to an existing file in Backblaze with a SHA-1 of only the 1 MByte chunk. That should allow you to stream? All great feedback, by the way. We really wa…

Adjustable chunk sizes would be beneficial as well.

Re: Backblaze B2 Cloud Storage

#184

Post by brianwski »

derefr wrote: post #160
brianwski wrote: post #155

Earlier quoted context omitted.

I think that's well put. I'm bothered by the whole idea of putting all my data with any one vendor (with Backblaze or Amazon) and thinking you don't need a backup. I claim "RAID / Reed-Solomon / real time mirrored copies" is NOT "Backup". If your programmer makes a mistake and a line of code deletes some mission critical data from Amazon S3, then all the Reed-Solomon encoding in the world doesn't help you, the data i…

> If your programmer makes a mistake and a line of code deletes some mission critical data from Amazon S3, then all the Reed-Solomon encoding in the world doesn't help you, the data is still gone. I'm surprised at the implication here, that you'd use Glacier on a non-versioned bucket. Making destructive updates impossible doesn't cost much extra in archive fees.

Ok, so let's say you forget to pay your Glacier bill because the IT guy left and the credit card changed and the alert emails go nowhere. Bye-Bye-Glacier! No payment, no customer data, Amazon might delete your data due to a tiny administrative screwup.

My point stands: if you don't mind losing your data, store it in one vendor. But if you would REALLY lose your business and put 10 people out of work if the data is lost, storing it in Amazon (or Backblaze) without a second copy backed up somewhere else and a third copy backed up in yet a third location (with a totally different vendor with a totally different payment system) is irresponsible.

Re: Backblaze B2 Cloud Storage

#185

Post by Dylan16807 »

jbeda wrote: post #147
heavenlyhash wrote: post #125

Earlier quoted context omitted.

> They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. Having been on the receiving end of entirely too many corrupted files in my life, I strongly approve of their use of a hash that's been standardized and fast for decades and remains cryptographically strong. "But fast" if you fail to store it isn't very helpful. TCP has a CRC too. We're wallpapering over it with better on…

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

A cryptographic hash is pretty much as fast as anything else, and lets you be certain. There's no good reason to use anything else.

Re: Backblaze B2 Cloud Storage

#186

Post by pfg »

kyledrake wrote: post #168
falsestprophet wrote: post #17

For context, OVH offers object storage with 3x replication for $.01/GB [1]. [1] https://www.runabove.com/cloud-storage.xml

I like this, but I've gotten mixed messages from the people running it. I tried to email them asking if they're actually working on the project and didn't get a response. I've also heard that they be migrating this to "OVH Public Cloud", which is a service that isn't available yet. I like this offering, but I'm not getting good signals on it's seriousness. It may be something they're going to sunset soon. I would nee…

Seems they're currently reorganising RunAbove and merging some of the projects back into the main OVH product line-up. OVH Public Cloud[1] is being rolled out at the moment, but not yet available in all regions.

[1]: https://www.ovh.co.uk/cloud/storage/object-storage.xml

Re: Backblaze B2 Cloud Storage

#187

Post by Dylan16807 »

atrophying wrote: post #149

Fair warning: Backblaze has a habit of making major changes silently (without any indication to the user), and their customer support is TERRIBLE. I was a customer for several years and never had a positive interaction with their support staff. The final straw was losing several files last year after they changed their backup method without notifying users, a method that contradicted their documentation. I got full I…

How did it change that you lost data?

Re: Backblaze B2 Cloud Storage

#188

Post by JungleGymSam »

Please tell me you're working on a PowerShell module. Microsoft is pushing PowerShell HARD (because it's awesome) and admins that do use it know its power and want ALL their vendors to use it.

You can open yourself up to a large number of customers by making it easy to get started via PowerShell.

Re: Backblaze B2 Cloud Storage

#190

Post by gabbo »

devit wrote: post #171
jbeda wrote: post #147

Earlier quoted context omitted.

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

No, it's pretty much totally unnecessary. The API works on top of TLS, which already includes cryptographic authentication of all data (usually via SHA-1/2 HMAC or AES-GCM). The hash would be computed at the client right after reading from disk and right before TLS enryption, and since they seem to terminate TLS at the storage server it would be computed right after TLS decryption and right before storage, so it does…

As jbeda mentions, hardware errors are one big reason: with the scale S3/Azure/GCS/Backblaze operate at it's a matter of when and not if you're going to run into problems. Also: TLS may guarantee the bits your client sends are the ones their server receives, but that's just one cause of errors.

There's the write path from B2 receives your bits to when they're stored on disk, for one. You could have unforeseen bugs in the code sitting on the other end of their upload URL (it's probably not all theirs, and even if it was it was written by human developers).

Or B2's internal network path (if they have any) between that and the disk. Ideally that would provide integrity too, but maybe not. They offer a low price point and call out other compromises they make to achieve it (e.g. limited load balancing) - so while I really doubt it, it's remotely plausible they deem the internal overhead of SSL too high.

But then there's the potential for mismatch between "what the customer thinks they uploaded" and "what the customer actually uploaded" too! Less of an issue for now because their API only appears to support uploading files all at once, but eventually I'm sure they'll support a multipart upload scheme like the other platforms do. At which point uploads become more complicated since clients need to retain state and potentially resume. What if a client screws it up and there's some off-by-one error (or whatever)? If you can provide instant feedback, at upload time, that your clients provided bogus data, that's a good thing.

You can argue it's a painful requirement to force on users since it means they have to track/compute it themselves (might be nontrivial for streaming applications), which is fair. But there are enough points of failure, and the numbers so large, that errors happening is a fact and you really need to insure against it. Especially here, your entire reason for existing is to reliably store bits so it's kinda important to get it provably right.

It seems completely sensible to err on the side of caution, especially as a new and relatively unproven platform (as an object storage platform provider I mean, obviously they have tons of experience storing things).

Post reply on HN