Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

backblaze.com

Post reply on HN
Print view View original

141–150 of 293 posts

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#141

Post by brianwski »

guenthert wrote: post #57
whalesalad wrote: post #49

This is great. Their current API requires you to identify a unique host to send data to, so you’re constantly performing a metric ton of DNS queries. Until I white listed the base domain it was the #1 client of my Pihole installation by multiple orders of magnitude.

The DNS resolver library of your client is allowed to cache the IP address for a given hostname for up to TTL. If it does so, the cost should be negligible.

Disclaimer: I work at Backblaze.

> The DNS resolver library of your client is allowed to cache the IP address for a given hostname for up to TTL

Not only that, but one mistake a lot of developers made early on was asking for a location to upload for every upload. That was NEVER the intention. In fact that annoys our servers also.

Developers are supposed to request a location to upload ONCE, and then upload to that location for hours, or even DAYS. Unless you have a bug in that software, it really shouldn't come anywhere close to being a high runner in DNS. We're talking 9 or 10 requests per day, at most, if you are unlucky. Feel free to reach out to our support if you aren't seeing that!

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#143

Post by hangonhn »

ksec wrote: post #38

Backblaze is also the founding member of Bandwidth Alliance, meaning getting those B2 via Cloudflare is essentially free. So you are only paying for Storage. ( Correct me if I am on this one ) I wonder why doesn't ALL non HyperScale Cloud Vendors, like Linode and DO provide one click third party backup to B2. You should always store offsite backup somewhere. And B2 is perfect.

This is so mind-blowing-ly crazy and awesome. I've always loved the idea of Backblaze but never tried it. But this combination is so insanely good and valuable. When an efficient market/economy works correctly, the results are just amazing: Cloudflare and Backblaze essentially got their unit costs for their respective products so low that for just a small money fee, I can do something that would have been prohibitively expensive maybe just a decade ago.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#144

Post by budmang »

heipei wrote: post #138
haywirez wrote: post #116

Earlier quoted context omitted.

I'm seeing horrible TTFB from Europe. Most files are fast but sometimes a request is stuck for 10s or more...

Is that to their US datacenter or the European (Amsterdam) one? So far the European one has been pretty snappy for me, ping is ~20ms from my home ISP (Germany) and TTFB is good enough that I can instantly saturate my 300MBit home ISP line with concurrency of ~ 500 downloading small objects (5b-1MB), but I haven't looked at min/max/avg/median yet. BIG gotcha btw: You have to choose between US and EU when you create yo…

(backblaze ceo here) Just fyi, you can put both accounts into a single Group for easier management: https://help.backblaze.com/hc/en-us/articles/115000014914

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#145

Post by atYevP »

TechBro8615 wrote: post #117
atYevP wrote: post #103

Earlier quoted context omitted.

For that integration it's just the download fees. The transactions do still get billed, but many people can stay within the free daily allotment.

Where can I find the pricing info for this allotment?

All of our pricing is available on the site (no need to call anyone) - B2 Cloud Storage is listed here -> https://www.backblaze.com/b2/cloud-storage-pricing.html. Just scroll down for the transactions!

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#146

Post by GeneticGenesis »

budmang wrote: post #136
benbro wrote: post #123

Is B2 suitable for streaming video files? Can I stream the same file to 1,000 viewers at the same time?

(backblaze ceo here) B2 is a great origin store for your video files. If you're streaming to lots of viewers, using a CDN with Backblaze B2 is optimal. We partnered with Cloudflare as a founding member of the Bandwidth Alliance so you can store your videos with B2 and transit them for free to Cloudflare, which can serve to your viewers.

Please correct me if I'm wrong, but my understanding was that Cloudflare should not be used to deliver video files unless using Cloudflare's "stream" product, IE specifically this [1].

[1]: https://community.cloudflare.com/t/cloudflare-how-not-to-vio...

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#147

Post by Dylan16807 »

derefr wrote: post #122
georgyo wrote: post #55

Earlier quoted context omitted.

It is, but they cap out the file size they will cache on the free plan to be 512MB. So you would need to chunk up your videos to get free bandwidth.

> So you would need to chunk up your videos to get free bandwidth. Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no? Is t…

> Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no?

Browsers are smart. They only buffer a few megabytes at a time and can seek around pretty efficiently.

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#148

Post by budmang »

guenthert wrote: post #68
mgamache wrote: post #4

S3 is now the standard for cloud storage APIs? Not sure if that's good or bad. I guess competitors have to reduce switching costs.

I don't see Google, MS Azure or Rackspace switching their API anytime soon, but yes, smaller players and new ones do.

(backblaze ceo here) Google and Azure continue to offer their own API as do we. They also offer S3 compatibility options to support customers who want to use S3-compatible products...as now do we ;-)

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#149

Post by heipei »

brianwski wrote: post #130
haywirez wrote: post #111

That's awesome, but I really want to see lightning fast response times and TTFB... Second pain point is the number of retries needed for uploading a large batch of small files. Those are the main reasons I'm still considering migrating away. I really wish I shouldn't as otherwise I love the pricing and the philosophy. Edit: also think DigitalOcean Spaces and B2 might be better off merging together, or Spaces being a…

Disclaimer: I work at Backblaze. > I really want to see lightning fast response times and TTFB (Time To First Byte Served) If a file is "cold" (nobody has requested it in the last 24 hours) then it needs to be reconstructed from the Backblaze Vaults and there is a little delay. After that, it should serve pretty fast for the following requests (off of a caching layer with SSDs). In the end, Backblaze B2 is a good sol…

I understand you're probably not in a position to say anything about it, but I'd love to see the "little delay" when reconstructing a file qualified somewhat. Are we talking < 5s or < 10s? What do the percentiles for restore latency look like? How does file size play into it? This, for me, is one of the biggest unknowns right now since it's not easy to create a test benchmark for this case (i.e. upload a bunch of stuff and let it sit idle for at least 24 hours, hoping it will be expired from the caching layer).

Re: Backblaze B2 Cloud Storage Now Has S3 Compatible APIs

#150

Post by derefr »

Dylan16807 wrote: post #147
derefr wrote: post #122

Earlier quoted context omitted.

> So you would need to chunk up your videos to get free bandwidth. Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no? Is t…

> Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no? Browsers are smart. They only buffer a few megabytes at a time and ca…

That requires the video to be encoded in a way where you can just start reading the stream from any random byte offset, and everything will still work. Video files are not usually encoded this way (any more.) Resume an MP4 or MKV video half-way through, without reading the TOC-ish stuff from the first chunk, and you'll get garbage that maybe resyncs after 20 seconds.

It's totally possible to "encode for streaming", but it usually results in both an increase in overhead [more keyframes] and a decrease in quality [inability to use predictive interpolation, instead relying only on forward-interpolation.]

Mind you, this streaming-enabled encoding is how things were done on the web, before the advent of MPEG-DASH/HLS; and it's still how e.g. the MP2 encoding of digital cable/satellite video works. But we don't really want to go back to those days. They kind of sucked.

Jumping to random byte offsets in a video also tends to screw with any embedded data streams like subtitles or thumbnails, which tend to just be stored in most media container formats as a single chunk at the beginning/end of the file, rather than being spread or copied across the stream. Again, the kind of captioning done back in the MP2 days is immune to this, but it kind of sucked as well (e.g. it wouldn't trigger if you happened to skip to the millisecond after the instruction for it appeared in the stream, often leaving you with ~30 seconds of untranslated audio.)

Post reply on HN