Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple downloads ~45 TB of models per day from our S3 bucket

twitter.com

Post reply on HN
Print view View original

211–220 of 237 posts

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#211

Post by cik »

bhaak wrote: post #153
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

I bet that 95% of all in-house CI would break if it doesn't have access to the internet. I also bet that 95% of those wouldn't need to have access if they were properly designed. We rarely hear about CI servers being taken over but it has to happen frequently enough.

You're absolutely right. It's the reason we always use devpi for python mirrors, and firewall off CI.

Ultimately you need to be able to rebuild the product when upstream goes poof.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#213

Post by pzmarzly »

MrStonedOne wrote: post #205
pzmarzly wrote: post #126

Earlier quoted context omitted.

> Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD Correct me if I'm wrong, but most colocation/dedicated server providers offer such prices. E.g. hetzner.com @ €1/TB, sprintdatacenter.pl @ €0.91/TB, dedicated.com @ $2/TB (or $600/month for an unmetered 1 Gbps connection). But if you want S3/CDNs/ , then yeah, they're expensive. BTW per Cloudflare ToS[0]: > Use of the Servi…

You confused cloudfront with cloudflare

True, I'm not that familiar with Amazon offerings (only ever used it for Windows GPU instances) and reading too fast got me. Too late for an edit.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#214

Post by guan »

simonh wrote: post #136
semi-extrinsic wrote: post #121

Earlier quoted context omitted.

The entire 32.0.0.0 used to be owned by a company that provided IT services to the Norwegian public sector. They had 4 IPs for every citizen in the country, and change.

Desktop, laptop, phone and tablet. Damn, ran out of IPs for the XBOX.

Network address, gateway, one usable IP address, broadcast address :-)

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#215

Post by kelnos »

JustFinishedBSG wrote: post #180
kelnos wrote: post #50

If a company the size of Apple finds this that useful, perhaps you should consider charging for your service, rather than just complaining on Twitter about the free usage you appear to have willingly given away? Or perhaps you have reached out to them, but are for some reason still complaining on Twitter to drum up PR or something? Regardless, this posting is ridiculously context-free to the point of being click-bait…

"It's your own fault if you didn't foresee a trillion dollar company exploiting the product you made free and open source to help researcher and now are incurring 15K$ bills per month" And then people wonder why people don't want to make their stuff free/open source. Even when it's free people still think you're somehow entitled and overcharging.

I mean... yes? It doesn't have to be a trillion dollar company. You put tons of useful data on a public S3 bucket and publicize it, and people are going to download it. S3 data transfer isn't free, so I think it's reasonable to expect that, over time, the cost of serving the data is going to be prohibitive without funding it in some way.

> Even when it's free people still think you're somehow entitled and overcharging.

I just explicitly advocated for the opposite of that, so I'm not sure where you're getting that.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#216

Post by rtkwe »

thenickdude wrote: post #162
rtkwe wrote: post #85

Earlier quoted context omitted.

That makes it harder for everyone though where companies like Apple should proxy and/or cache those requests to their own internal version rather than hitting that S3 bucket every time. Requiring requester payment would mean it would only really be used by corporations where the author clearly wants a service open to anyone without having to open an AWS account to pay.

Make the bucket buyer-pays, but offer Torrent links as well. Businesses doing CI will pay for S3 usage so they don't have to deal with torrents, end-users will get free torrent access, everybody wins.

The S3 bucket is a repository for the models which get downloaded at runtime it seems so a torrent isn't really a good fit for a drop in replacement.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#217

Post by fs111 »

baroffoos wrote: post #97
CobrastanJorji wrote: post #56

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into…

Or just post a magnet link.

torrents were supported by S3 in the past https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#218

Post by rtkwe »

baroffoos wrote: post #97
CobrastanJorji wrote: post #56

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into…

Or just post a magnet link.

That probably wouldn't work here. That s3 bucket is hosting models downloaded at runtime/startup [1] and looks like under normal runs it would be cached. If this is being used in Apple's CI pipeline though the whole thing is being torn down between builds so every build and test has to fetch it again.

[1] https://github.com/huggingface/pytorch-transformers/search?q...

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#219

Post by rrix2 »

MrStonedOne wrote: post #204
Fradow wrote: post #183

Earlier quoted context omitted.

Not hard to find: https://eugdprcompliant.com/personal-data/ state "The conclusion is, all IP addresses should be treated as personal data, in order to be GDPR compliant." Obviously different jurisdiction have a different notion of PII. We agree on the conclusion though: a public range for a company probably doesn't count as PII.

Is that page legally binding? Does it cite anything legally binding? No? Then it's not a source. >The GDPR states that IP addresses should be considered personal data as it enters the scope of ‘online identifiers’. Of course, in the case of a dynamic IP address – which is changed every time a person connects to a network – there has been some legitimate debate going on as to whether it can truly lead to the identific…

Sources: Article 4(1) and Recital 30.

Either way Apple is not a Person with data rights in this scenario.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#220

Post by kalleboo »

treypitt wrote: post #210
kalleboo wrote: post #77

Earlier quoted context omitted.

This map was drawn 13 years ago so it's heavily out of date but it does illustrate the companies that got their ip /8 blocks back the day (Ford?!) https://xkcd.com/195/

Won't every vehicle soon have its own IP address? Many must have them already. IoT means some devices need multiple IP addresses

Ford was assigned it's netblock in 1988. It was definitely not assigned with IoT in mind. And if they deployed IoT today, they would just use the mobile telco's dynamic IP addresses and communicate through HTTP.
Post reply on HN