Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

backblaze.com

Post reply on HN
Print view View original

251–260 of 293 posts

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#251

Post by janjanson »

derefr wrote: post #150
Dylan16807 wrote: post #147

Earlier quoted context omitted.

> Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no? Browsers are smart. They only buffer a few megabytes at a time and ca…

That requires the video to be encoded in a way where you can just start reading the stream from any random byte offset, and everything will still work. Video files are not usually encoded this way (any more.) Resume an MP4 or MKV video half-way through, without reading the TOC-ish stuff from the first chunk, and you'll get garbage that maybe resyncs after 20 seconds. It's totally possible to "encode for streaming", b…

I don't think this is quite right. If you serve an mp4 with H.264 video statically on any basic webserver it will just work in browser through plain HTTP, without need need for MPEG-DASH/HLS. Every widely used mediaplayer/browser just downloads the nearest chunk (keyframe point) behind the time that was seeked to can resume the playback from there. This point is found through an index stored in a container format. For basically every video format these days (say, at least as new as H.264), regular settings make this only a few more seconds of video to download and decode before the seeked point and basically happens instantly for normal online consumption. In H.264 forward prediction (through two-pass encoding) will playback fine too.

I think what you're saying applies more to a setting where the video is being streamed live, so that you cannot access the start of the file to get keyframe metadata. In that case HLS and MPEG-DASH help.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#252

Post by waffle_ss »

sime2009 wrote: post #72
waffle_ss wrote: post #43

Earlier quoted context omitted.

That's a fair point but this is for my family photos and videos; in the event I die I'd rather the S3 bucket I hand over to my wife/kids in my last wishes look like a real filesystem rather than a bazillion blobs that require a special tool with a programmer's expertise to reassemble

May I ask which solution you ended up going for in this situation? (I'm trying to solve for myself too.)

I have a local ZFS pool of hard drives and a script that `rsync -avz`s my iPhone's photos+videos to it. Then a separate cron script that periodically syncs from the ZFS pool to my S3 bucket using `aws s3 sync`. The S3 bucket has versioning turned on so it's effectively append-only.

I used to be able to trigger my iPhone -> ZFS script when I plugged my iPhone into my Ubuntu desktop using udev (also had to wrap it in flock[1] because it would trigger multiple times for some reason), but at some point that stopped working and I've been too lazy to figure out why.

It's far from perfect but for me it works alright. In this scenario I prefer straightforward and slightly kludgy compared to something with hidden complexity that could go wrong in so many ways. Could you imagine if you used a tool like restic or borg and the pack encoding format changed, or if the tool sources are simply gone when your relatives have to figure out how to get at the files in 10, 15, 20 years - I don't want my relatives playing code detective or archaeologist!

Which reminds me of a downside to using tools like restic, borg, and the like I forgot to mention. When I evaluated them for my hundreds of GB of family pics+videos, there is a "dedupe" step that all these tools want to perform. When I tested them a couple years ago they were dog slow for my files, because pics + video are already highly compressed and there is very little "deduping" you're going to wring out of them unless you have multiple copies of the same files. IIRC borg took several hours to run at at the end it reported 0.01% or less deduping efficiency. Also as I recall there was no way to opt-out of the dedupe step due to the way borg stores "packs". Very annoying!

[1]: https://stackoverflow.com/a/169969/215168

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#253

Post by yjftsjthsd-h »

jjeaff wrote: post #87
tambre wrote: post #13

Earlier quoted context omitted.

>It requires Amazon to have a massive and expensive choke point in their network: load balancers. When a customer tries to upload to S3, she is given a single upload URL to use. For instance, s3.amazonaws.com/ . Now that Amazon has deprecated the single URL version and replaced it with region-specific URLs (e.g. s3.dualstack.us-east-1.amazonaws.com) and tooling has been mostly updated, this huge reason for not suppor…

Even though they are using region specific URLs, they would still have to load balance all of that traffic. Backblaze avoided this by having a 2 part request for a file. You would make a request to a centralized url and then that would return a url that connected you directly to a server that had that file.

> You would make a request to a centralized url and then that would return a url that connected you directly to a server that had that file.

Heh, kinda like how FTP worked. That's funny to see again.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#254

Post by ADefenestrator »

toohotatopic wrote: post #169
ksec wrote: post #160

Earlier quoted context omitted.

On the note of unit cost I doubt it will be much lower in the near future. Cost / GB isn't falling much if at all, and their Load balancer are already eating into their margin which means any further reduction in unit cost are only gaining back what they took in with the S3 API. Hopefully the new S3 compatible API means they could make all that back with volume. But this is also a sad realisation that storage tech ar…

Wasn't there a startup on hn some weeks ago that used a crypto mining approach to server management that was much cheaper?

Disclaimer - Backblaze employee here, but just speaking for myself:

It was.. sort of cheaper. They didn't actually build the servers, and as described the server wouldn't work (onboard SATA didn't support port multipliers, lack of ECC would probably cause problems in practice, bit hand-wavey on power/space/network/manpower costs, etc). The goal of the article was to get other people to build cheap storage and put it up for rent on their network. They do have some amount of storage space available for very cheap on the network now, but personally I suspect it's people who figured "what the heck, I'll give it a try!" as opposed to people actually building storage servers and making a profit renting them out.

I was honestly pretty disappointed - I'd hoped they'd found a cheap motherboard with ECC and support for port multipliers, but nope.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#255

Post by ADefenestrator »

neurostimulant wrote: post #191
jorams wrote: post #107

Earlier quoted context omitted.

Can you use process substitution? b2 upload_file bucket The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.

Does process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.

It definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#257

Post by brainless »

anotheracct_ wrote: post #163
atYevP wrote: post #58

Earlier quoted context omitted.

Yev from Backblaze here -> That's correct. We're a founding member of the Bandwidth Alliance and you can find more info about that on our Blog ( https://www.backblaze.com/blog/backblaze-and-cloudflare-part... ) or FAQ ( https://help.backblaze.com/hc/en-us/articles/217666928-Using... ). May the 4th be with you ;-)

Isn't this incompatible with Net Neutrality? I think Cloudflare already was against net neutrality, but people who believe in that principle might need to avoid Backblaze as well if that is the case.

Airlines have alliances, it does not stop you from taking any airline you want for your route, but costs may vary.

It is just a property of networks. As an end user you still get to use all providers, but two of them may use a dedicated connection to reduce internal prices (the price benefit of which they may or may not give back to the user).

Neutrality from the point of the user is not affected when it comes to service being available.

Now if one airline alliance said you are not allowed to get on their flight if you hoped off from a rival alliance, then we have an issue.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#258

Post by PhantomGremlin »

rsync wrote: post #164
waffle_ss wrote: post #43

Earlier quoted context omitted.

That's a fair point but this is for my family photos and videos; in the event I die I'd rather the S3 bucket I hand over to my wife/kids in my last wishes look like a real filesystem rather than a bazillion blobs that require a special tool with a programmer's expertise to reassemble

" ... in the event I die I'd rather the S3 bucket I hand over to my wife/kids in my last wishes look like a real filesystem rather than a bazillion blobs that require a special tool ..." You'd need a cloud storage provider that just gave you a plain old UNIX filesystem to do whatever you want with. It's too bad nobody does that ...

You'd need a cloud storage provider that just gave you a plain old UNIX filesystem to do whatever you want with.

Doesn't iCloud Drive fit the bill?

Access to it is slightly obscure, i.e. ~/Library/Mobile Documents/com~apple~CloudDocs/ but wouldn't that work?

It's free for me to use. That's because I'm already paying Apple $10/mo to backup the family's iPhones. We're only using a little over 200 GB out of the 2000 GB we have. (I'm sure that Apple is counting on most people not using their full amount).

I've only put a few files out there, so maybe there are a lot of potential pitfalls. But it doesn't get much simpler than using cp or mv.

In reality it's most emphatically not a "plain old UNIX filesystem". Apple is doing some magic and storing blobs out in Amazon S3 or in their own datacenters. But to me it has the appearance of a Unix (Posix?) filesystem.

I realize that rsync.net couldn't survive with a business model that limits users to 2000 GB, which is Apple's maximum. But I thought I'd mention it, since it just might be the perfect "free" solution for a lot of people.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#259

Post by bvivek77 »

budmang wrote: post #100
rochester46 wrote: post #74

Earlier quoted context omitted.

Does AWS (and GCP and Azure) not do the exact same thing when transferring data out via Cloudfront (or their respective CDN)? Is Backblaze really special in that regard?

(backblaze ceo here) AWS is free to Cloudfront because that's an AWS service. AWS/GCP/Azure are not free (and quite expensive) to Cloudflare. Backblaze is free to Cloudflare.

Bit confusing. Need more clarity. Read 10 times. Pls recheck mr. ceo :)
Post reply on HN