[deleted]
Apple downloads ~45 TB of models per day from our S3 bucket
81–90 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#82A brief reminder: Whenever you publish code or documentation that might be used/scraped by the outside world, ALWAYS use a domain you own. If you're on Cloudflare you can instantly (and for free) create Page Rules to use Cloudflare as a CDN, redirect to another CDN, or black-hole or reroute traffic anywhere you want.
When working on APIs meant to be used client-side (especially mobile clients) by different customers are partners, use one subdomain per integrator. If there's a bug in their integration, it could easily DDoS your servers, but DNS is an easy way to have a manual kill switch.
Literally had this happen to us (website, not API) from a misconfigured partner last week - they accidentally misrouted unrelated click traffic through our servers. A 2 minute Page Rule and we not only saved our servers, we protected our partner's brand until they could hotfix. We could have done this with a rule looking at the path, but not easily something looking at obscure auth keys. Segmented traffic is happy traffic.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#83Have you considered cloudflair?
Backblaze B2 + Cloudflare = free outbound transfer (bandwidth alliance). Making objects in S3 publicly available and then complaining about their extortionate bandwidth charges is...silly.
Cloudflare will cut you off and require a paid plan before you get to 1/10th this amount of egress.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#84If a company the size of Apple finds this that useful, perhaps you should consider charging for your service, rather than just complaining on Twitter about the free usage you appear to have willingly given away? Or perhaps you have reached out to them, but are for some reason still complaining on Twitter to drum up PR or something? Regardless, this posting is ridiculously context-free to the point of being click-bait…
On the flip side huge corporations like Apple should be more careful about not abusing free services set up by people not just because it's a dick thing to drown free services in traffic just because they're free but also it's a pretty big security issue downloading (seemingly at runtime given how much they're downloading each day) from a random bucket you don't control.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#85Hosting terabytes of data on an S3 bucket where people would download 45TB per month ($0.023/GB == $1000+/month) sounds like a really expensive way to distribute your data to people...
You could configure buyer pays, if you wanted to.
That makes it harder for everyone though where companies like Apple should proxy and/or cache those requests to their own internal version rather than hitting that S3 bucket every time. Requiring requester payment would mean it would only really be used by corporations where the author clearly wants a service open to anyone without having to open an AWS account to pay.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#86Earlier quoted context omitted.
16.7 million or so. IPv4 is a 32 bit address space, so it tops out around 4.2 billion total. 17.0.0.0/8 is locking down the first 8 bits, giving 2^(32-8) variable bits, or there are only 256 possible first octets and this is one so it’s 1/256th of 4.2 billion addresses.
It’s even more when you factor in link local and private network ranges.
Everybody has those for themselves.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#87Earlier quoted context omitted.
They're basically bragging they have something Apple really wants. Now they have a bunch of people at least interested in what they got. I'll say that's not a bad PR.
Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.
There is no expectation of anything without a paid contract. This isn't a major issue though, no enterprise is going to care about their team freely downloading generic public ML data from the internet. If they did care, they would already have other arrangements.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#88"Almost everyone" working on NLP uses one of hugginface's pretrained models at one point or another, sooner or later: https://github.com/huggingface/pytorch-transformers It's so damn convenient, and so nicely done. And they keep doing neat things like this one: https://github.com/huggingface/swift-coreml-transformers Kudos to Julien Chaumond et al for their work!
> Swift Core ML
Why does he call them Swift Core ML? They are core ml, usable in swift and objective-c.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#89Earlier quoted context omitted.
Backblaze B2 + Cloudflare = free outbound transfer (bandwidth alliance). Making objects in S3 publicly available and then complaining about their extortionate bandwidth charges is...silly.
Cloudflare will cut you off and require a paid plan before you get to 1/10th this amount of egress.
Still cheaper than AWS!
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#90Earlier quoted context omitted.
Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
It is public those are Apple’s IPs. It is not public they are downloading data from this S3 bucket, never mind how much.