Apple downloads ~45 TB of models per day from our S3 bucket • Hacker News Board

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#181

Post by oblio » Tue, Sep 17, 2019, 9:48 AM UTC

takeda wrote: post #118

ethbro wrote: post #79

Earlier quoted context omitted.

Map: https://www.caida.org/research/id-consumption/census-map/ima... (a bit dated, but shows the historic allocations) Look up "CIDR", history thereof, for reasons why it looks this way.

Crazy how wasteful it is. I wonder what genius thought to allocate /8 to every company/organization. You don't need to have PhD in statistics and math to know there's more than 250 companies.

Well, same thing for the geniuses that made IPs 32 bit when even MACs are 48 bit.

Or, if my networking trainer at a Cisco course is to be believed, the geniuses that made IPv6 subnets contains 65k hosts at a minimum, when due to ARP requests all traffic would be dead at that scale.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#182

Post by MertsA » Tue, Sep 17, 2019, 9:50 AM UTC

ms4720 wrote: post #168

takeda wrote: post #124

Earlier quoted context omitted.

well since DoD started the whole Internet thingy that's not that surprising, but yeah they were extra wasteful.

Loop back address, 127.0.0.1, is really a class A block, 127.0.0.0/8, that only uses 1 address

You can use anything from that block as well and there are situations where you might want a different loopback address or multiple loopback addresses. /8 of the public address space is massively excessive though.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#183

Post by Fradow » Tue, Sep 17, 2019, 9:50 AM UTC

paranoidrobot wrote: post #175

distant_hat wrote: post #167

Earlier quoted context omitted.

IP addresses are definitely PII under GDPR.

Do you have a source for that? My understanding is that it is only PII if it's in conjunction with other data in particular ways. When the GDPR came along and IP Addresses were being mentioned as PII, my employer required us to sign a document stating (in part) that we wouldn't access, download or communicate PII data except when specifically authorised to do so. When I refused to sign that and ran it up the flagpole…

Not hard to find: https://eugdprcompliant.com/personal-data/ state "The conclusion is, all IP addresses should be treated as personal data, in order to be GDPR compliant."

Obviously different jurisdiction have a different notion of PII.

We agree on the conclusion though: a public range for a company probably doesn't count as PII.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#184

Post by [deleted] » Tue, Sep 17, 2019, 9:50 AM UTC

owenmarshall wrote: post #116

flotwig wrote: post #74

Earlier quoted context omitted.

GE actually sold their entire 3.0.0.0/8 block off to AWS a few years ago. It's a little awkward since a lot of internal software is still configured to whitelist all access from that space since it was a constant for so long.

We called this threxit internally. (Get it? Three dot exit? ;)) And as far as I know we haven’t stopped threxiting — at least they hadn’t when I left. It turns out unwinding IT systems that have had stable IP addresses for 30+ years in a year or two is tricky business.

[deleted]

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#185

Post by corint » Tue, Sep 17, 2019, 9:54 AM UTC

Hitton wrote: post #177

thenickdude wrote: post #162

Earlier quoted context omitted.

Make the bucket buyer-pays, but offer Torrent links as well. Businesses doing CI will pay for S3 usage so they don't have to deal with torrents, end-users will get free torrent access, everybody wins.

I just skimmed AWS requester pays and it seems that you can't set the price, the requester always pays only the AWS's download cost, so why bother setting it as requester pays at all if it doesn't bring you anything?

It brings you the absence of GET requests and the bandwidth charges on your AWS bill. In this case, 45TB a day is going to add a lot to an AWS bill, you can shift that cost to the user that's downloading the files from you.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#186

Post by amelius » Tue, Sep 17, 2019, 9:58 AM UTC

anaisbetts wrote: post #163

coldtea wrote: post #150

Earlier quoted context omitted.

So they can fix it, now, that is is a problem, and they didn't have to spend time worrying about that before. Sounds very smart on their party: move forward with what matters (building your codebase, tests, etc) and don't do something (like an internal cache), unless you have to -- the ops equivalent of lazy-loading...

It's smart but pretty fucking rude tbh.

It's not that smart because they might throttle the bandwidth as a result. Also, one day the downloading may take longer than the time between tests and the entire test will start failing. Further, if you want to test the testing procedure, you want a cache anyway.

The data should probably be distributed using e.g. git, so that downloads are incremental by default.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#187

Post by ChrisRR » Tue, Sep 17, 2019, 10:14 AM UTC

klohto wrote: post #138

viraptor wrote: post #27

Earlier quoted context omitted.

You don't know that this was the only action he took. This was not a great criticism considering how limited our information is right now.

He literally said he blocked their IP range

The comment said only action. He blocked their IP range but probably contacted them too.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#188

Post by goatinaboat » Tue, Sep 17, 2019, 10:22 AM UTC

bryanrasmussen wrote: post #159

dharmon wrote: post #15

I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?

if one of the richest companies in the world is hammering your server without paying for it I would hope they don't get their feelings hurt when the server blocks them until they pay for it. I mean I've worked at some of the riches companies in the world and I think the conversation would have gone like this Me: hey project manager our access to server X where we get the really needed X1 resource has been blocked. Pr…

Can you tell them we are trying to get a paid account, see if you can get them to give us temporary access, and maybe we can cache the result so we don't hit their server that often. Me: ok I'll do that

I’m not sure I would feel comfortable doing that - stringing along a fellow engineer for the benefit of a freeloading PM.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#189

Post by koolba » Tue, Sep 17, 2019, 10:26 AM UTC

manigandham wrote: post #86

koolba wrote: post #65

Earlier quoted context omitted.

It’s even more when you factor in link local and private network ranges.

Everybody has those for themselves.

That’s what I’m saying. The set of addresses allocated to Apple (in their /8) is an even larger ratio as the private ranges don’t count towards the denominator.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#190

Post by asymptotically2 » Tue, Sep 17, 2019, 10:27 AM UTC

simonh wrote: post #136

semi-extrinsic wrote: post #121

Earlier quoted context omitted.

The entire 32.0.0.0 used to be owned by a company that provided IT services to the Norwegian public sector. They had 4 IPs for every citizen in the country, and change.

Desktop, laptop, phone and tablet. Damn, ran out of IPs for the XBOX.

You could (and should) do that with IPv6. Down with NAT! :)