Live data from Hacker News

Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

backblaze.com

191–200 of 293 posts

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#191
post #107

Earlier quoted context omitted.

I really wish the B2 client support uploading file from Unix pipe. It would be nice to be able to archive a huge directory into a tar.bz archive and directly pipe the result into the B2 client without having to save the archive into disk first. Currently I have to save the tar.bz archive to disk first before uploading to balance. Took several hours to do so (huge spinning disks array, not as fast as ssd), while uploa…

Can you use process substitution? b2 upload_file bucket The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.

Does process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#193
post #180

Earlier quoted context omitted.

I'm referring to Cloudflare as the network provider. It seems not-net-neutral to be charged different amounts depending on the relationship between the storage provider and the network provider. If the argument is a semantic one, that net neutrality is so narrowly defined that it doesn't apply to them as a CDN rather than a conventional ISP, it still stands that the situation is not-net-neutral if we generalize that…

cloudflare is not charging different amount for different storage providers, some storage providers are waiving the fee when the upstream is cloudflare. disclaimer: ex-cloudflare

So, Backblaze is not-net-neutral.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#194
post #150

Earlier quoted context omitted.

> Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no? Browsers are smart. They only buffer a few megabytes at a time and ca…

That requires the video to be encoded in a way where you can just start reading the stream from any random byte offset, and everything will still work. Video files are not usually encoded this way (any more.) Resume an MP4 or MKV video half-way through, without reading the TOC-ish stuff from the first chunk, and you'll get garbage that maybe resyncs after 20 seconds. It's totally possible to "encode for streaming", b…

That's odd. I can't recall ever having a problem with browsers playing mp4s in a vanilla tag, as long as I encode them in the main h264 profile, AAC audio, and MOOV atom at the front (see [0] for ffmpeg command). Obviously the server has to support Range: byte requests.

My impression is DASH/HLS are mostly useful for adjusting bitrate on the fly.

[0]: https://superuser.com/a/438471/402047

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#195

Earlier quoted context omitted.

No, this has nothing to do with net neutrality. Your ISP isn't discriminating what data you consume, whether it's from cloudflare or another CDN. This is about the fact that bandwidth is usually bought by capacity between networks rather than usage and they're passing those savings on to you instead of charging expensive per-GB costs like the major clouds.

I'm referring to Cloudflare as the network provider. It seems not-net-neutral to be charged different amounts depending on the relationship between the storage provider and the network provider. If the argument is a semantic one, that net neutrality is so narrowly defined that it doesn't apply to them as a CDN rather than a conventional ISP, it still stands that the situation is not-net-neutral if we generalize that…

Network neutrality is a difficult concept to apply because its core concept doesn't map well to reality.

The core concept, in my mind, is that all packets should be treated equally, and should cost the customer the same amount per byte.

The problem is that different packets take different network paths and have different costs to your provider. The bandwidth alliance partners are exchanging traffic via peering links, so there's only the fixed costs of equipment and any port charges/interconnection fees to the peering facility. Packets sent to other destinations may pass through a paid transit link and cost the provider. If your provider charges you the same amount for both types of packets, they're not being transparent about their network costs which is bad for you, and is also bad for them, because you may be able to adjust your traffic so more is settlement free and less is paid --- but you won't do that without an incentive.

Of course, if you don't like how Backblaze manages its network, there's a bunch of other network storage vendors who you could switch to (many of which are members of the Bandwidth Alliance). Or, it's not too hard to build a storage box or two and get them into a colocation somewhere.

For your residential ISP though, for most people, if you don't like the network policies, you might have another option, but they will likely have similar policies. You don't have meaningful choice in a single residence, and moving residences to get better choices isn't a meaningful option either. So, network neutrality is a policy to (attempt to) regulate residential ISP behavior, in order to provide a reasonable policy for users. It's still a problem that it doesn't match reality, and it doesn't provide user choice, but it has gotten a lot of support.

I would rather see mandatory line sharing for residential ISPs, it's a lot easier to define and with proper regulation offers a path towards consumer choice, and gives people a way to take action if their provider has poor network policies --- if your provider runs an acceptable last mile service, but provides poor interconnection to the rest of the world, you could become a line sharing partner and provide better interconnection to the world without having to build out an overlay last-mile network (which is incredibly capital intensive and difficult)

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#196
post #179

Earlier quoted context omitted.

Isn't this incompatible with Net Neutrality? I think Cloudflare already was against net neutrality, but people who believe in that principle might need to avoid Backblaze as well if that is the case.

Yev here -> unsure how those are related, but you certainly don't NEED to use Cloudflare if you're using B2 Cloud Storage as the data origination point, it's just an option that was brought up in the thread!

It sounded like the price would be different if you used a certain network provider, which is not-net-neutral.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#197
post #164

Earlier quoted context omitted.

That's a fair point but this is for my family photos and videos; in the event I die I'd rather the S3 bucket I hand over to my wife/kids in my last wishes look like a real filesystem rather than a bazillion blobs that require a special tool with a programmer's expertise to reassemble

" ... in the event I die I'd rather the S3 bucket I hand over to my wife/kids in my last wishes look like a real filesystem rather than a bazillion blobs that require a special tool ..." You'd need a cloud storage provider that just gave you a plain old UNIX filesystem to do whatever you want with. It's too bad nobody does that ...

I was just about to pull out my rsync.net pitchfork before I saw the username. You should throwaway next time so as to not deprive me of the pleasure.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#198
post #54

Earlier quoted context omitted.

What region are you moving from/to? Last I checked, b2 only exists in datacenters on the US west coast.

(backblaze ceo here) We also have a region in Europe: https://www.backblaze.com/blog/announcing-our-first-european...

Please consider an Asia/Pacific data center. I am from India and my company was not able to use B2 due to high response times even from European DC. Even a DC in Singapore will be helpful for us.

- Thankful Personal Backup Customer

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#199
post #173

Used B2 heavily until recently as a origin server for a CDN. Few weeks ago we saw a spike in 502 / 504 responses. When I contacted their customer supported , I was pointed to the following URL where they explain in detail how they handle these errors https://www.backblaze.com/blog/b2-503-500-server-error/ Essentially they are not considered as errors and expect the client to retry loading the file. This approach won'…

If you are using Cloudflare you could use a Worker script to automatically retry the origin pull and only cache it on success.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#200
post #8

Earlier quoted context omitted.

So how did they manage to get rid of those hidden costs? Or is the new S3 compatible API more expensive?

Disclaimer: I work at Backblaze. > So how did they manage to get rid of those hidden costs? Or is the new S3 compatible API more expensive? The new S3 compatible APIs are the same cost as the original native B2 APIs. I'm the author of that original blog post, and we were able to get rid of SOME of the internal costs, but in the end we ate the cost of the load balancer. Internally I voted that we externalize that cost…

Are you guys apprehensive at all that supporting S3 might bring in more lower margin customers (ie higher transfer GB/store GB ratio)?

In any case, thanks for B2 (happy customer) and good luck. Sounds like an exciting time.

Post reply on HN