Isn’t this a use case where BitTorrent would shine?
Apple downloads ~45 TB of models per day from our S3 bucket
131–140 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#132That's about $4000/month in bandwidth costs, assuming retail pricing. FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.
> That's about $4000/month in bandwidth costs You're an order of magnitude off. 45 TB per day is 1,350 TB in a month, or 1,350,000 GB. Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD... Let's suppose you even used the cheaper egress from Cloudfront rather than serving from S3 (lol @ your wallet if you serve 1 PB doing that). https://aws.amazon.com/blogs/aws/aws-data-transf…
I may be a little off, but definitely not by an order of magnitude. Even high-end hosting at a provider like Akamai can be under $10/TB-mo for such large volumes. And since you don't need any of the hundred value-added services that Akamai, Cloufront, Fastly etc. offer, the budget providers can go a lot lower than that.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#133Earlier quoted context omitted.
I used to work for HP and someone explained to me a select few companies got /8's when the internet was still young. HP got one, Compaq had one which HP now also owns. I was basically told if you had a /8 you didn't give it up because of how valuable and rare they now are (this was around 2010, too). GE, Kodak, Apple, and Microsoft were a few other names that came up in that discussion as well.
Funny enough the highest price point for IP ranges is somewhere between /16 and /24, IIRC. You can count how many companies need and will be willing to pay 8-9 figures for a /8 without getting to your toes. And subnetting it and selling it to maximize returns is hard work. But if you’re sitting on a /21? That’ll move before you can count how many IPs are in the block ;)
Why are they so valuable?
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#134Earlier quoted context omitted.
Not to mention that if Cloudflare CDN was in front of it this traffic would be free.
Not with this amount of traffic it won't be free. 45 TB per day lol Cloudflare will be disabling your account and in contact for payment in a hurry. Go ahead and try, see how far their "free" tier really goes.
Not sure about that. Especially if it all goes to one ip address where they have peering arrangements. It could cause some load there but the traffic will be essentially free for cloudflare. And some good publicity for Cloudflare.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#135If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into…
Or use Cloudflare in front of it, especially with mostly static data.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#136Earlier quoted context omitted.
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
The entire 32.0.0.0 used to be owned by a company that provided IT services to the Norwegian public sector. They had 4 IPs for every citizen in the country, and change.
Desktop, laptop, phone and tablet. Damn, ran out of IPs for the XBOX.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#137If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into…
Or use Cloudflare in front of it, especially with mostly static data.
Cloudflare generally doesn’t want to cache data files on non enterprise plans (the terms
generally disallow it). Only websites HTML and related content.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#138I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?
You don't know that this was the only action he took. This was not a great criticism considering how limited our information is right now.
He literally said he blocked their IP range
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#139This is probably apple's continuous integration tests, lazily written to download the whole thing every time someone merges a commit.
that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository.
In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#140Earlier quoted context omitted.
Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.
Apple publishes that IP range (CIDR address block) in several KB articles on its own website for system administrators to configure firewalls/web filters. https://support.apple.com/en-us/HT210060 https://support.apple.com/en-gb/HT203609
That's rather beside the point. Just because I advertise my IP doesn't mean I want my service providers to disclose my activity publicly.