That's about $4000/month in bandwidth costs, assuming retail pricing.
FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.
Apple downloads ~45 TB of models per day from our S3 bucket
51–60 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#52I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?
You're joking right? Lots of people use twitter. It's not just for tweens anymore. This is as good a way as any to get Apple to take notice and maybe send some bucks their way. It's a half joke/half serious attempt
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#53Earlier quoted context omitted.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
It's a /8, which is 2^(32 - 8) = 16.7 million address.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#54Earlier quoted context omitted.
Apple employees are using their product, downloading lots of data, not paying for any of it, and the OP doesn't like it or can afford it.
I don't think it's employees as such — even Apple does not have THAT many machine learning people, and they wouldn't download models daily. Maybe a server farm, where each instance downloads a model when spinning up?
It sounds like a misconfigured build pipeline which re-downloads the model for every single build.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#55Have you considered cloudflair?
Nobody is going to regularly let you have 45 terabytes of data for free. It's swamping them, especially relative to other users.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#56If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves.
This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into a program, plus everyone who wants your dataset needs to set up a billing account with Amazon or Google.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#57Earlier quoted context omitted.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
Apple has a /8 of IPv4 space (plus a couple other small allocations: https://whois.arin.net/rest/org/APPLEC-1-Z/nets ), which contains 16777216 addresses: https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#58A brief reminder: Whenever you publish code or documentation that might be used/scraped by the outside world, ALWAYS use a domain you own. If you're on Cloudflare you can instantly (and for free) create Page Rules to use Cloudflare as a CDN, redirect to another CDN, or black-hole or reroute traffic anywhere you want.
Not to mention that if Cloudflare CDN was in front of it this traffic would be free.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#59I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?
That tweet will get attention! Finding the right person is everything...
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#60Earlier quoted context omitted.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
16.7 million or so.
IPv4 is a 32 bit address space, so it tops out around 4.2 billion total.
17.0.0.0/8 is locking down the first 8 bits, giving 2^(32-8) variable bits, or there are only 256 possible first octets and this is one so it’s 1/256th of 4.2 billion addresses.