Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple downloads ~45 TB of models per day from our S3 bucket

twitter.com

Post reply on HN
Print view View original

111–120 of 237 posts

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#111

Post by angrygoat »

hinkley wrote: post #68
microtherion wrote: post #18

Earlier quoted context omitted.

I don't think it's employees as such — even Apple does not have THAT many machine learning people, and they wouldn't download models daily. Maybe a server farm, where each instance downloads a model when spinning up?

In the last days of my time spent in the XML salt mines, I got in on a conversation with the web masters at w3.org. You would not believe how many people and how many libraries pull from primary sources directly instead of using local copies of common resources. I found this conversation because I'd just finished fixing that in our code and taking about 5 minutes off the build process. Let me restate that: We were sp…

I sometimes wonder about all the additional HTTP load to places like debian.org that must come from Docker builds; I don't have any kind of caching in mine, and every single commit causes CI to go and build images and run tests on them.

It's not an issue I really see, but it seems to me that CI infrastructure (Travis, ...) really should have caching proxies in place.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#113

Post by arthurfm »

joatmon-snoo wrote: post #32
httpz wrote: post #25

Earlier quoted context omitted.

They're basically bragging they have something Apple really wants. Now they have a bunch of people at least interested in what they got. I'll say that's not a bad PR.

Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.

Apple publishes that IP range (CIDR address block) in several KB articles on its own website for system administrators to configure firewalls/web filters.

https://support.apple.com/en-us/HT210060

https://support.apple.com/en-gb/HT203609

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#114

Post by [deleted] »

stjohnswarts wrote: post #52
dharmon wrote: post #15

I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?

You're joking right? Lots of people use twitter. It's not just for tweens anymore. This is as good a way as any to get Apple to take notice and maybe send some bucks their way. It's a half joke/half serious attempt

[deleted]

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#116

Post by owenmarshall »

flotwig wrote: post #74
brodie78382 wrote: post #67

Earlier quoted context omitted.

I used to work for HP and someone explained to me a select few companies got /8's when the internet was still young. HP got one, Compaq had one which HP now also owns. I was basically told if you had a /8 you didn't give it up because of how valuable and rare they now are (this was around 2010, too). GE, Kodak, Apple, and Microsoft were a few other names that came up in that discussion as well.

GE actually sold their entire 3.0.0.0/8 block off to AWS a few years ago. It's a little awkward since a lot of internal software is still configured to whitelist all access from that space since it was a constant for so long.

We called this threxit internally. (Get it? Three dot exit? ;))

And as far as I know we haven’t stopped threxiting — at least they hadn’t when I left. It turns out unwinding IT systems that have had stable IP addresses for 30+ years in a year or two is tricky business.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#117

Post by owenmarshall »

brodie78382 wrote: post #67
vonseel wrote: post #48

Earlier quoted context omitted.

Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?

I used to work for HP and someone explained to me a select few companies got /8's when the internet was still young. HP got one, Compaq had one which HP now also owns. I was basically told if you had a /8 you didn't give it up because of how valuable and rare they now are (this was around 2010, too). GE, Kodak, Apple, and Microsoft were a few other names that came up in that discussion as well.

Funny enough the highest price point for IP ranges is somewhere between /16 and /24, IIRC.

You can count how many companies need and will be willing to pay 8-9 figures for a /8 without getting to your toes. And subnetting it and selling it to maximize returns is hard work.

But if you’re sitting on a /21? That’ll move before you can count how many IPs are in the block ;)

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#118

Post by takeda »

ethbro wrote: post #79
brodie78382 wrote: post #67

Earlier quoted context omitted.

I used to work for HP and someone explained to me a select few companies got /8's when the internet was still young. HP got one, Compaq had one which HP now also owns. I was basically told if you had a /8 you didn't give it up because of how valuable and rare they now are (this was around 2010, too). GE, Kodak, Apple, and Microsoft were a few other names that came up in that discussion as well.

Map: https://www.caida.org/research/id-consumption/census-map/ima... (a bit dated, but shows the historic allocations) Look up "CIDR", history thereof, for reasons why it looks this way.

Crazy how wasteful it is. I wonder what genius thought to allocate /8 to every company/organization. You don't need to have PhD in statistics and math to know there's more than 250 companies.

Re: Apple downloads ~45 TB of models per day from our S3 bucket

#120

Post by CobrastanJorji »

baroffoos wrote: post #97
CobrastanJorji wrote: post #56

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves. This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into…

Or just post a magnet link.

Sure, that's a great option for helping to reduce the cost for well-meaning general use, but the other way makes your costs 100% predictable, which is great if you're on an academic budget (but, again, way more annoying for the downloaders unless they're also using AWS).
Post reply on HN