Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple downloads ~45 TB of models per day from our S3 bucket

twitter.com

Post reply on HN
Print view View original

151–160 of 237 posts

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#151

Post by coldtea »

Manfred wrote: post #148
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

Unfortunately it's very common for companies to set up their CI without any form of caching. I think it's mostly because developers are under time pressure from their managers. In some cases it's because CI is set up by juniors who don't fully understand the tools and the consequences of setting them up at this scale.

It's also not a problem until it is. Then, you can devote resources at it, but meanwhile you got things up and running for months/years faster than if you tried to get everything setup just so from the get go...

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#152

Post by Yizahi »

joatmon-snoo wrote: post #32
httpz wrote: post #25

Earlier quoted context omitted.

They're basically bragging they have something Apple really wants. Now they have a bunch of people at least interested in what they got. I'll say that's not a bad PR.

Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.

Oh, so now IP addresses is PII? When it is inconvenient for FAAGM monster corporations? I seem to remember a few hundred thousands corporate statements that tracking individual IPs is totally ok and not surveillance.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#153

Post by bhaak »

mister_hn wrote: post #139
nurettin wrote: post #119

This is probably apple's continuous integration tests, lazily written to download the whole thing every time someone merges a commit.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

I bet that 95% of all in-house CI would break if it doesn't have access to the internet. I also bet that 95% of those wouldn't need to have access if they were properly designed.

We rarely hear about CI servers being taken over but it has to happen frequently enough.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#155

Post by dageshi »

joatmon-snoo wrote: post #32
httpz wrote: post #25

Earlier quoted context omitted.

They're basically bragging they have something Apple really wants. Now they have a bunch of people at least interested in what they got. I'll say that's not a bad PR.

Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.

If an enterprise user cares about privacy and non-disclosure they'd have a contract with the guy providing the service surely? If they don't... then they really don't care about privacy and non-disclosure that much.

Apple obviously didn't.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#156

Post by donkeyd »

Yizahi wrote: post #152
joatmon-snoo wrote: post #32

Earlier quoted context omitted.

Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.

Oh, so now IP addresses is PII? When it is inconvenient for FAAGM monster corporations? I seem to remember a few hundred thousands corporate statements that tracking individual IPs is totally ok and not surveillance.

In the Netherlands an IP address is legally PII. I'm not sure this is true for publicly available IP ranges, especially if they're owned by companies though (not like a person would own a range), but probably not.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#157

Post by falsedan »

coldtea wrote: post #150
mister_hn wrote: post #139

Earlier quoted context omitted.

that's really stupid. I mean, I would have set a cache repository (SonaType Nexus maybe?), download everything there and use that repository. In the tweets the author says they've blocked the download from Apple IPs, so now their pipeline is broken.

So they can fix it, now, that is is a problem, and they didn't have to spend time worrying about that before. Sounds very smart on their party: move forward with what matters (building your codebase, tests, etc) and don't do something (like an internal cache), unless you have to -- the ops equivalent of lazy-loading...

Every medium-sized org I’ve worked at has put a caching layer in front of their build dependencies, so builds aren’t blocked when GitHub/PyPI are unavailable. No build/release engineer would leave that trivial door open if they were responsible for the build

Sounds like the build pipeline was set up by a regular dev.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#158

Post by sleepychu »

lolc wrote: post #149
OGWhales wrote: post #133

Earlier quoted context omitted.

Why are they so valuable?

Because there are only four billion addresses all in all. Every device that wants to be reachable on the Internet needs one. These days, mobile devices don't get a public address anymore and there are all sorts of complications due to it. We're slowly transitioning to a new scheme with ample address space. But to nobody's surprise it's taking decades longer than envisioned.

Hopefully it's surprising to the envisioner(s)!

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#159

Post by bryanrasmussen »

dharmon wrote: post #15

I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?

if one of the richest companies in the world is hammering your server without paying for it I would hope they don't get their feelings hurt when the server blocks them until they pay for it.

I mean I've worked at some of the riches companies in the world and I think the conversation would have gone like this

Me: hey project manager our access to server X where we get the really needed X1 resource has been blocked. Probably cause we are hammering their server (which we probably talked about some months ago)

Project manager: hmm a paid account will need to go through approval. Can you tell them we are trying to get a paid account, see if you can get them to give us temporary access, and maybe we can cache the result so we don't hit their server that often.

Me: ok I'll do that.

In fact I mean if I was on a small company providing a service that a really big company was abusing in such a way that I think they might like to pay for it - I think the smart thing would be to force them to contact me somehow because how do I even call the part of the big company I need to talk to.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#160

Post by martin-adams »

StreamBright wrote: post #144

Paid by requester is the feature they are looking for. https://docs.aws.amazon.com/AmazonS3/latest/dev/configure-re...

That's a very neat feature I never knew existed. I only suspect this will hurt their larger mission at helping many smaller teams and individuals to use the models.
Post reply on HN