Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze B2 Cloud Storage

backblaze.com

Post reply on HN
Print view View original

231–237 of 237 posts

Re: Backblaze B2 Cloud Storage

#231

Post by brianwski »

hobs wrote: post #217
brianwski wrote: post #177

Earlier quoted context omitted.

Brian from Backblaze here. I wonder if that was during the incredibly annoying "Comcast goes to war with Netflix" era that Backblaze got caught up in. That was Nov 2013 through Feb 2014, you can read a little about it here: https://www.backblaze.com/blog/obama-backs-net-neutrality/ (scroll down for our graphs showing our customers getting throttled). That seriously sucked for Backblaze. But either way, we added threa…

Brian, thank you for responding, I appreciate your clarity and honesty. My issue did occur during that period, and I am impressed you can call that out from memory, it must have been a frustrating time for BB. If that alone was the problem, you would have just 100% won back a customer, but the thing that irked me the most was the customer support response. I know you do not work for your helpdesk, but their response…

I can laugh about that period now, but yeah, it was a bad few months. We bled out good customers like yourself and we felt helpless. My basic faith in the internet was shaken up - I always thought I would send packets and they would be delivered quickly, and here are these HUGE players in the space messing with each other throttling each other and changing routing to get around throttling (and hurting Backblaze as collateral damage).

> do not work for your helpdesk

It's unfortunate when a customer gets a bad experience. The helpdesk guys are faced with this monumental task of responding to tons and tons of basic questions by Mom & Pop customers that are not computer professionals. Then mixed in are competent programmers and IT guys that know what the heck they are talking about. The helpdesk guys sometimes get it wrong who they are dealing with and it infuriates the competent computer users.

I think we should issue "professional computer user" cards where you can get a different level of support from all these companies. If you were helpful on forums you could earn points for your card, but if you ask helpdesk too many dumb questions your card could be revoked and you would go back to the first tier support. :-)

Re: Backblaze B2 Cloud Storage

#232

Post by brianwski »

voltagex_ wrote: post #196
brianwski wrote: post #167

Earlier quoted context omitted.

A python command line tool that should work on Linux to send folders to B2 is available for download here: https://www.backblaze.com/b2/docs/quick_command_line.html CAVEAT (PLEASE READ): this does NOT encrypt data yet!! This is just a quick technology demonstration, it isn't a polished backup client. Give us another month for that... We expect Linux servers (and desktops) to make up a significant percentage of the th…

If I was going to tar and then gpg files, what would be the optimal "chunk size" from your point of view? Say I have 10GB that I'm encrypting and then splitting up, are you going to want 1GB chunks or many more 100mb chunks? (This also raises the question of how much data I want to lose to corruption...)

I would recommend 1 GByte chunks for several reasons. The first reason is that you'll get very good throughput and efficiency at that large size. If you go as small as 10 MBytes we see long distances to our datacenter not "ramping up" fully. In other words, we can't get a single threaded application between New Zealand and California to get the advertised bandwidth for small chunks, but up at 1 GByte we DO get the advertised bandwidth. But if you go too large (like 5 GBytes) we have seen that you start seeing too high a percentage of uploads rejected because of bit errors occurring somewhere in the networking between your computer and our servers (the SHA-1 checksum won't match).

Re: Backblaze B2 Cloud Storage

#233

Post by ak217 »

jbeda wrote: post #147
heavenlyhash wrote: post #125

Earlier quoted context omitted.

> They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. Having been on the receiving end of entirely too many corrupted files in my life, I strongly approve of their use of a hash that's been standardized and fast for decades and remains cryptographically strong. "But fast" if you fail to store it isn't very helpful. TCP has a CRC too. We're wallpapering over it with better on…

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

The CRC in TCP is not powerful enough, but CRCs can be adjusted to be arbitrarily powerful. The main advantage of CRCs is that they can be independently computed for multiple parts and combined when concatenating the parts.

Re: Backblaze B2 Cloud Storage

#234

Post by ak217 »

heavenlyhash wrote: post #125
jbeda wrote: post #38

Took a quick look at the API. For context, I was involved in the early days of Google Cloud Storage. It is surprising that they didn't make it compatible with the S3 API -- at least for common object/bucket create/delete. This will require more code to be written and it will be harder to adapt client libraries. The API documentation is here: https://www.backblaze.com/b2/docs/ Other notes: * The lack of scalable front…

> They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. Having been on the receiving end of entirely too many corrupted files in my life, I strongly approve of their use of a hash that's been standardized and fast for decades and remains cryptographically strong. "But fast" if you fail to store it isn't very helpful. TCP has a CRC too. We're wallpapering over it with better on…

The CRC in TCP is weak. CRCs can be made arbitrarily powerful and still retain their key advantage: they can be computed independently and then combined when concatenating the parts. That key advantage, combined with a strong CRC, is what provides an elegant solution to the 2-pass problem.

Re: Backblaze B2 Cloud Storage

#235

Post by ak217 »

devit wrote: post #171
jbeda wrote: post #147

Earlier quoted context omitted.

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

No, it's pretty much totally unnecessary. The API works on top of TLS, which already includes cryptographic authentication of all data (usually via SHA-1/2 HMAC or AES-GCM). The hash would be computed at the client right after reading from disk and right before TLS enryption, and since they seem to terminate TLS at the storage server it would be computed right after TLS decryption and right before storage, so it does…

There are many places in your stack where data corruption can and will occur. You are correct that TLS provides payload integrity on a per-packet basis - but it doesn't protect you against silent truncation (to fight this, always declare and check content-length, or use chunked encoding). I have seen corruption occur in NIC buffers, ECC'd main memory, Xen MMU'd memory pages (yes, Xen was responsible), and multiple places in HTTP server and client stacks. None of those failures manifested until hundreds of terabytes of data had successfully gone through the system.

If you're handling data on behalf of others, it's paramount that you checksum data end-to-end. Amazon S3 allows you to do this by sending the MD5 or SHA along with the data. Google GCE allows you to do this with CRCs (which, despite what others in this thread say, are more appropriate for the task than crypto hashes, as long as you use enough bits).

Re: Backblaze B2 Cloud Storage

#236

Post by atrophying »

Dylan16807 wrote: post #187
atrophying wrote: post #149

Fair warning: Backblaze has a habit of making major changes silently (without any indication to the user), and their customer support is TERRIBLE. I was a customer for several years and never had a positive interaction with their support staff. The final straw was losing several files last year after they changed their backup method without notifying users, a method that contradicted their documentation. I got full I…

How did it change that you lost data?

They started picking and choosing which parts of AppData they backed up, but never informed their users and kept on claiming they backed up "everything." See, the backup app used to be set up so you could see what was being excluded, and modify it if need be. I had the entirety of AppData selected as "for backup," then had several bits of saved user data wiped out. One of those was an extension that stored its data in the browser user profile, something I used frequently and had restored a half-dozen times from backup. Then it was gone. Backblaze stopped backing it up, deleted it from their servers, and I had no way to restore it. Three years of data lost and zero effort from Backblaze to recover it, not even an apology.

The technology isn't bad, but their customer service is some of the worst I've ever seen. I was a Backblaze customer for three years and not once did I have what I'd consider a positive experience. If anything goes wrong they leave you hanging. They're not a company I'd ever trust with valuable data again.

Post reply on HN