And now the twitter post is gone? I'm guessing the west coast woke up and someone at Apple said "Wait, you could infer some proprietary information with that information ..."
Apple downloads ~45 TB of models per day from our S3 bucket
201–210 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#202And now the twitter post is gone? I'm guessing the west coast woke up and someone at Apple said "Wait, you could infer some proprietary information with that information ..."
I can see the Twitter post without issues.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#203And now the twitter post is gone? I'm guessing the west coast woke up and someone at Apple said "Wait, you could infer some proprietary information with that information ..."
I can see the Twitter post without issues.
Yeah, the follow_up after clearing some caches it seems to work for me as well. Interesting.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#204Earlier quoted context omitted.
Do you have a source for that? My understanding is that it is only PII if it's in conjunction with other data in particular ways. When the GDPR came along and IP Addresses were being mentioned as PII, my employer required us to sign a document stating (in part) that we wouldn't access, download or communicate PII data except when specifically authorised to do so. When I refused to sign that and ran it up the flagpole…
Not hard to find: https://eugdprcompliant.com/personal-data/ state "The conclusion is, all IP addresses should be treated as personal data, in order to be GDPR compliant." Obviously different jurisdiction have a different notion of PII. We agree on the conclusion though: a public range for a company probably doesn't count as PII.
Is that page legally binding?
Does it cite anything legally binding?
No? Then it's not a source.
>The GDPR states that IP addresses should be considered personal data as it enters the scope of ‘online identifiers’. Of course, in the case of a dynamic IP address – which is changed every time a person connects to a network – there has been some legitimate debate going on as to whether it can truly lead to the identification of a person or not. The conclusion is that the GDPR does consider it as such.
>.... The conclusion is that the GDPR does consider it as such.
How?
The article just spews word vomit and then makes a conclusion on behalf of the GDPR without even citing a single bit of the GDPR to back up its arguments.
You should never take legal advise from a website that doesn't cite the legal text.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#205Earlier quoted context omitted.
> That's about $4000/month in bandwidth costs You're an order of magnitude off. 45 TB per day is 1,350 TB in a month, or 1,350,000 GB. Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD... Let's suppose you even used the cheaper egress from Cloudfront rather than serving from S3 (lol @ your wallet if you serve 1 PB doing that). https://aws.amazon.com/blogs/aws/aws-data-transf…
> Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD Correct me if I'm wrong, but most colocation/dedicated server providers offer such prices. E.g. hetzner.com @ €1/TB, sprintdatacenter.pl @ €0.91/TB, dedicated.com @ $2/TB (or $600/month for an unmetered 1 Gbps connection). But if you want S3/CDNs/ , then yeah, they're expensive. BTW per Cloudflare ToS[0]: > Use of the Servi…
You confused cloudfront with cloudflare
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#206Earlier quoted context omitted.
That's a very neat feature I never knew existed. I only suspect this will hurt their larger mission at helping many smaller teams and individuals to use the models.
Not really, if you are a small user your cost is negligible. You can calculate how much it would be for a small team but my guess is couple of dollars per month. Apple's use case is still very reasonable and the cost for them also not as bad. They could also split out different customers to different buckets and have big guys pay for it while smaller companies have it for free. There are many options.
Yeah that makes sense. I was thinking in terms of ease of access. If a large organisation makes everyone pay, that means everyone has to arrange accounts and payment methods.
By splitting it out you have no guarantee that the big guys will just use the free one.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#207Earlier quoted context omitted.
Yes. I don’t know if that’s all that they do. They often port new Tensorflow models to PyTorch as well. They provide straightforward APIs, nice documentation, and clear tutorials. I use their stuff pretty regularly.
Do you know their business model? Looks like they are open source company. How they earn money?
Their business model is someone like Apple buying them out.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#208Earlier quoted context omitted.
Crazy how wasteful it is. I wonder what genius thought to allocate /8 to every company/organization. You don't need to have PhD in statistics and math to know there's more than 250 companies.
I wonder who will think about the genius who disbanded the EPA, rolled back every environmental protection there is and withdrew from the Paris Agreement at the most critical time for our planet in 40 years. Hindsight is 20/20 and the „Internet“ was a mainly US centered university research project that was thought of as a toy by the far majority. Everybody thought they‘d have a replacement for the initial assignment…
Wouldn't the more apt comparison be to the folks who set up the EPA in the first place (Nixon administration I believe)?
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#209This looks kind of interesting: https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... When you look further down you find: https://github.com/huggingface/pytorch-pretrained-BigGAN/blo... And that's just a quick search for s3 in the repo. It would not surprise me in the least to discover a `from_pretrained` that points at one of the s3 resources being pulled. There's probably other stuff like that as well in…
It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.
Docker builds are crazy wasteful in terms of bandwidth and compute. Right now I'm struggling with a project that builds ITK on demand every heckin time.
I'm working through how to best integrate apt-cacher-ng, sccache, and a pip cacher. It costs me nothing to hit apt or pypi, but like, somebody is paying that bill. A little perspective goes a long way.
I wonder if I could do something to just proxy all requests and cache those on a whitelist and stick it on my CI network.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#210Earlier quoted context omitted.
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
This map was drawn 13 years ago so it's heavily out of date but it does illustrate the companies that got their ip /8 blocks back the day (Ford?!) https://xkcd.com/195/
Won't every vehicle soon have its own IP address? Many must have them already. IoT means some devices need multiple IP addresses