Apple downloads ~45 TB of models per day from our S3 bucket • Hacker News Board

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#221

Post by Fradow » Tue, Sep 17, 2019, 2:51 PM UTC

MrStonedOne wrote: post #204

Fradow wrote: post #183

Earlier quoted context omitted.

Not hard to find: https://eugdprcompliant.com/personal-data/ state "The conclusion is, all IP addresses should be treated as personal data, in order to be GDPR compliant." Obviously different jurisdiction have a different notion of PII. We agree on the conclusion though: a public range for a company probably doesn't count as PII.

Is that page legally binding? Does it cite anything legally binding? No? Then it's not a source. >The GDPR states that IP addresses should be considered personal data as it enters the scope of ‘online identifiers’. Of course, in the case of a dynamic IP address – which is changed every time a person connects to a network – there has been some legitimate debate going on as to whether it can truly lead to the identific…

You are not going to find anything in the GDPR that cleary state if IP are personal data or not, because the GDPR is tech-agnostic and thus doesn't use tech-specific terminology such as IP.

Article 4 [1] states > ‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person;

Can a person be identified by an IP, which is an online identifier? That's up to a judge to decide, according to context. Some judges do think so (French)[2]. The interesting part there: "Les adresses IP collectées [...] doivent par ailleurs être considérées comme une collecte à grande échelle de données d’infraction au sens de l’article 10 du RGPD". Rough translation : "Collected IP adresses [...] must be considered as large scale collection of offenses data in terms of article 10 of the GDPR" (article 10 is about personal data for offenses and convictions).

You can find a few similar cases where IPs are considered as personal data. It has been a subject of discussion here on HN several times during the GDPR introduction.

Please note that I never said or implied that any of what I said or linked is legal advice or a legal text.

[1] https://gdpr.eu/article-4-definitions/ [2] https://www.legalis.net/jurisprudences/tgi-de-paris-ordonnan...

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#222

Post by coleca » Tue, Sep 17, 2019, 3:17 PM UTC

z3t4 wrote: post #154

Apple are probably doing "continuous integration" where all assets are re-downloaded from the Internet in each iteration. Tip: put your stuff on Github :P

With models that large you would be paying for GitHub's LFS credits. Those aren't cheap as I recall. Napkin math at $5/50GB of bandwidth per month for 45TB per day it would cost them $135k/mo to use Github. That's over 2x more than the S3 egress charges would be.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#223

Post by mst » Tue, Sep 17, 2019, 3:46 PM UTC

bhaak wrote: post #153

mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

I bet that 95% of all in-house CI would break if it doesn't have access to the internet. I also bet that 95% of those wouldn't need to have access if they were properly designed. We rarely hear about CI servers being taken over but it has to happen frequently enough.

Fairly sure that was the vector for the big matrix.org breach.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#224

Post by joecot » Tue, Sep 17, 2019, 4:04 PM UTC

zenexer wrote: post #36

jrullman wrote: post #14

Earlier quoted context omitted.

You can use Cloudflare Workers to rewrite the path.

Those get fairly expensive if you have a lot of requests. Page rules are free if you only need a few, though.

3 pages rules are free. You need at least 2 page rules per website you want to use with Backblaze and Cloudflare, and that's for bare minimum workability.

For a commercial company paying $5-10 a month to save TBs of bandwidth via Backblaze & Cloudflare is a no brainer. It's just unfortunately not very workable for small projects. Github Pages works with Cloudflare for static sites just fine though.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#225

Post by bryanrasmussen » Tue, Sep 17, 2019, 4:20 PM UTC

goatinaboat wrote: post #188

bryanrasmussen wrote: post #159

Earlier quoted context omitted.

if one of the richest companies in the world is hammering your server without paying for it I would hope they don't get their feelings hurt when the server blocks them until they pay for it. I mean I've worked at some of the riches companies in the world and I think the conversation would have gone like this Me: hey project manager our access to server X where we get the really needed X1 resource has been blocked. Pr…

Can you tell them we are trying to get a paid account, see if you can get them to give us temporary access, and maybe we can cache the result so we don't hit their server that often. Me: ok I'll do that I’m not sure I would feel comfortable doing that - stringing along a fellow engineer for the benefit of a freeloading PM.

in the scenario I was envisioning, the big rich company wants to get a paid account now that they need it but to do so they will have to go through a process to get it approved which could take a while.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#226

Post by takeda » Tue, Sep 17, 2019, 5:06 PM UTC

ethbro wrote: post #199

takeda wrote: post #118

Earlier quoted context omitted.

Crazy how wasteful it is. I wonder what genius thought to allocate /8 to every company/organization. You don't need to have PhD in statistics and math to know there's more than 250 companies.

Maybe take a dose of humility and realize that at one point the fastest processors and memory systems in the world weren't capable of holding more than a limited size routing table, while maintaining acceptable line speed? And that in the interests of working within the physical hardware limitations of the day, very smart engineers made the best choices they could? Jesus.

What this has to do with anything? I'm saying that giving whole /8 (or I should say class A) to a company is wasteful. And you can only do it no more than a bit over 200 times. You are on the other hand saying that the hardware at the time wouldn't be able to handle all the companies. Why not allocate C blocks, or at very least B blocks? Or are you saying that they doubted hardware of the future would be capable of handling it?

Because of such wasteful allocation we got this "wonderful" thing called NAT which basically killed most of innovation in area of networking and IPv6 which is taking over 20 years to adapt, because most ISPs hold to IPv4 as long as they can because making this switch requires some work.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#227

Post by [deleted] » Tue, Sep 17, 2019, 5:09 PM UTC

paxys wrote: post #51

That's about $4000/month in bandwidth costs, assuming retail pricing. FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.

[deleted]

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#228

Post by ShteiLoups » Tue, Sep 17, 2019, 5:27 PM UTC

BlueGh0st wrote: post #191

cs702 wrote: post #40

"Almost everyone" working on NLP uses one of hugginface's pretrained models at one point or another, sooner or later: https://github.com/huggingface/pytorch-transformers It's so damn convenient, and so nicely done. And they keep doing neat things like this one: https://github.com/huggingface/swift-coreml-transformers Kudos to Julien Chaumond et al for their work!

For anyone else initially confused, NLP in this context is "Natural Language Processing."

As opposed to?

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#229

Post by hinkley » Tue, Sep 17, 2019, 5:41 PM UTC

angrygoat wrote: post #111

hinkley wrote: post #68

Earlier quoted context omitted.

In the last days of my time spent in the XML salt mines, I got in on a conversation with the web masters at w3.org. You would not believe how many people and how many libraries pull from primary sources directly instead of using local copies of common resources. I found this conversation because I'd just finished fixing that in our code and taking about 5 minutes off the build process. Let me restate that: We were sp…

I sometimes wonder about all the additional HTTP load to places like debian.org that must come from Docker builds; I don't have any kind of caching in mine, and every single commit causes CI to go and build images and run tests on them. It's not an issue I really see, but it seems to me that CI infrastructure (Travis, ...) really should have caching proxies in place.

Several places I've worked have pushed Artifactory on us for various reasons from security to resilience to network outages to bandwidth saving.

It's not the worst fate.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#230

Post by enjoy-your-stay » Tue, Sep 17, 2019, 5:58 PM UTC

dogecoinbase wrote: post #57

vonseel wrote: post #48

Earlier quoted context omitted.

Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?

Apple has a /8 of IPv4 space (plus a couple other small allocations: https://whois.arin.net/rest/org/APPLEC-1-Z/nets ), which contains 16777216 addresses: https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...

Well that should just be about enough for each customer on Apple iWeb Cloud Services to have their own ip address once it gets started.. :')