That's about $4000/month in bandwidth costs, assuming retail pricing. FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.
Could be a lucrative way of eliminating that problem.
Apple downloads ~45 TB of models per day from our S3 bucket
61–70 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#62I'm guessing someone at apple internally distributed a dockerfile that pulls that down.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#63This looks kind of interesting: https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... When you look further down you find: https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... And that's just a quick search for s3 in the repo. It would not surprise me in the least to discover a `from_pretrained` that points at one of the s3 resources being pulled. There's probably other stuff like that as well in…
It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#64Hosting terabytes of data on an S3 bucket where people would download 45TB per month ($0.023/GB == $1000+/month) sounds like a really expensive way to distribute your data to people...
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#65Earlier quoted context omitted.
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
16.7 million or so. IPv4 is a 32 bit address space, so it tops out around 4.2 billion total. 17.0.0.0/8 is locking down the first 8 bits, giving 2^(32-8) variable bits, or there are only 256 possible first octets and this is one so it’s 1/256th of 4.2 billion addresses.
It’s even more when you factor in link local and private network ranges.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#66Hosting terabytes of data on an S3 bucket where people would download 45TB per month ($0.023/GB == $1000+/month) sounds like a really expensive way to distribute your data to people...
You could configure buyer pays, if you wanted to.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#67Earlier quoted context omitted.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
I used to work for HP and someone explained to me a select few companies got /8's when the internet was still young. HP got one, Compaq had one which HP now also owns. I was basically told if you had a /8 you didn't give it up because of how valuable and rare they now are (this was around 2010, too). GE, Kodak, Apple, and Microsoft were a few other names that came up in that discussion as well.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#68Earlier quoted context omitted.
Apple employees are using their product, downloading lots of data, not paying for any of it, and the OP doesn't like it or can afford it.
I don't think it's employees as such — even Apple does not have THAT many machine learning people, and they wouldn't download models daily. Maybe a server farm, where each instance downloads a model when spinning up?
In the last days of my time spent in the XML salt mines, I got in on a conversation with the web masters at w3.org.
You would not believe how many people and how many libraries pull from primary sources directly instead of using local copies of common resources. I found this conversation because I'd just finished fixing that in our code and taking about 5 minutes off the build process.
Let me restate that: We were spending 5 minutes just downloading schema files. In an automated build. Every time, sometimes on several machines at once.
At one point we were trying to convince him that intentionally slowing all requests down by say 500 ms would get the attention of people who were misbehaving. Anyone who was downloading it once and caching would hardly notice the 500 ms. Those running it once per task would be forced to figure their shit out.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#69A brief reminder: Whenever you publish code or documentation that might be used/scraped by the outside world, ALWAYS use a domain you own. If you're on Cloudflare you can instantly (and for free) create Page Rules to use Cloudflare as a CDN, redirect to another CDN, or black-hole or reroute traffic anywhere you want.
Not to mention that if Cloudflare CDN was in front of it this traffic would be free.
I'm skeptical of the number of 500+ MB files the CloudFlare CDN would actually cache...
Does anyone have any numbers on this?
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#70Earlier quoted context omitted.
Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
[deleted]