Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple downloads ~45 TB of models per day from our S3 bucket

twitter.com

Post reply on HN
Print view View original

161–170 of 237 posts

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#161

Post by Aissen »

lolc wrote: post #145
black_puppydog wrote: post #143

Earlier quoted context omitted.

Maybe I'm misunderstanding something here, but does apple pay them for this?

The point of this thread is that maybe Apple would pay if they were asked.

And how would you be contacting Apple, from your little 3-person startup in Paris ? You assume they have the means or contacts to do that; and IMHO the tweet is not that aggressive.

It's been done before, (e.g Intel has been called out for consuming kernel.org bandwidth and git CPU power) and is the simplest way to have people from inside BigCorp get a message.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#162

Post by thenickdude »

rtkwe wrote: post #85
xfitm3 wrote: post #66

Earlier quoted context omitted.

You could configure buyer pays, if you wanted to.

That makes it harder for everyone though where companies like Apple should proxy and/or cache those requests to their own internal version rather than hitting that S3 bucket every time. Requiring requester payment would mean it would only really be used by corporations where the author clearly wants a service open to anyone without having to open an AWS account to pay.

Make the bucket buyer-pays, but offer Torrent links as well. Businesses doing CI will pay for S3 usage so they don't have to deal with torrents, end-users will get free torrent access, everybody wins.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#163

Post by anaisbetts »

coldtea wrote: post #150
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

So they can fix it, now, that is is a problem, and they didn't have to spend time worrying about that before. Sounds very smart on their party: move forward with what matters (building your codebase, tests, etc) and don't do something (like an internal cache), unless you have to -- the ops equivalent of lazy-loading...

It's smart but pretty fucking rude tbh.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#164

Post by peterwwillis »

If your CI/CD is re-downloading and re-building everything on every single run, you are not only being wasteful, you're actually more likely to have an outage due to not storing dependency artifacts needed for deploy. Use a local artifact store to be more resilient to failures of servers you don't control (and also save everyone money and time).

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#165

Post by lolc »

sleepychu wrote: post #158
lolc wrote: post #149

Earlier quoted context omitted.

Because there are only four billion addresses all in all. Every device that wants to be reachable on the Internet needs one. These days, mobile devices don't get a public address anymore and there are all sorts of complications due to it. We're slowly transitioning to a new scheme with ample address space. But to nobody's surprise it's taking decades longer than envisioned.

Hopefully it's surprising to the envisioner(s)!

I assume it was willful optimism on the part of the envisioners. If you tell people it will take decades it will take even longer!

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#166

Post by lolc »

Aissen wrote: post #161
lolc wrote: post #145

Earlier quoted context omitted.

The point of this thread is that maybe Apple would pay if they were asked.

And how would you be contacting Apple, from your little 3-person startup in Paris ? You assume they have the means or contacts to do that; and IMHO the tweet is not that aggressive. It's been done before, (e.g Intel has been called out for consuming kernel.org bandwidth and git CPU power) and is the simplest way to have people from inside BigCorp get a message.

No need to argue with me. I'm only explaining the original point, not making it myself :-)

But reviewing the thread I see you your question was in response to someone calling them "service providers" to Apple. So your question was entirely justified and it was me who'd lost context.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#167

Post by distant_hat »

Yizahi wrote: post #152
joatmon-snoo wrote: post #32

Earlier quoted context omitted.

Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.

Oh, so now IP addresses is PII? When it is inconvenient for FAAGM monster corporations? I seem to remember a few hundred thousands corporate statements that tracking individual IPs is totally ok and not surveillance.

IP addresses are definitely PII under GDPR.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#168

Post by ms4720 »

takeda wrote: post #124
xxpor wrote: post #76

Earlier quoted context omitted.

The DOD has like 5 or 6!

well since DoD started the whole Internet thingy that's not that surprising, but yeah they were extra wasteful.

Loop back address, 127.0.0.1, is really a class A block, 127.0.0.0/8, that only uses 1 address

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#169

Post by nurettin »

coldtea wrote: post #150
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

So they can fix it, now, that is is a problem, and they didn't have to spend time worrying about that before. Sounds very smart on their party: move forward with what matters (building your codebase, tests, etc) and don't do something (like an internal cache), unless you have to -- the ops equivalent of lazy-loading...

I was asked to scrape some prediction website millions of times every day. Instead of doing that, I reverse engineered their prediction curves from a few hundred data points and served the models myself. Not sure which one is more ethical. Scraping at scale or stealing the models using mathematics. But I know second one is cooler.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#170

Post by mickeyp »

bhaak wrote: post #153
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

I bet that 95% of all in-house CI would break if it doesn't have access to the internet. I also bet that 95% of those wouldn't need to have access if they were properly designed. We rarely hear about CI servers being taken over but it has to happen frequently enough.

Most enterprises use Artifactory or something similar -- or should anyway. Once you start enforcing "no internet for CI" you start to see how poor some ecosystems are. I'm looking at you, Javascript ecosystem packages, with your hardcoded mystery URLs that you sneakily download artefacts from...
Post reply on HN