Earlier quoted context omitted.
This isn't sensitive information. Anyone with a BGP session can have this information. [1] [1] = https://bgp.he.net/AS714#_prefixes
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
Apple downloads ~45 TB of models per day from our S3 bucket
121–130 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#122Earlier quoted context omitted.
Pretty amazing that a single company can own an entire block of IP space, if I understand this correctly. Approx how many addresses is this?
This map was drawn 13 years ago so it's heavily out of date but it does illustrate the companies that got their ip /8 blocks back the day (Ford?!) https://xkcd.com/195/
BTW: this one is better: https://www.caida.org/research/id-consumption/census-map/ima...
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#123Earlier quoted context omitted.
Correct; Cloudflare doesn't cache large asset files (I think anything more than 2MB?) by default. It's not that kind of CDN... at least, not for free it's not. Of course, you can trick Cloudflare into caching your large media assets using some funky Page Rules... but I wouldn't suggest it. Mostly just for moral reasons. If you have that much traffic, you should be making some money off it and then paying Cloudflare w…
512mb: https://support.cloudflare.com/hc/en-us/articles/200172516-U... And a "cache everything" page rule tends to cache literally any file type, but it's not a great idea to push media files through CF due to the TOS prohibiting "disproportionate amounts of non-web content". 2mb might be referencing the limit for Workers KV.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#124Re: Apple downloads ~45 TB of models per day from our S3 bucket
#125Re: Apple downloads ~45 TB of models per day from our S3 bucket
#126That's about $4000/month in bandwidth costs, assuming retail pricing. FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.
> That's about $4000/month in bandwidth costs You're an order of magnitude off. 45 TB per day is 1,350 TB in a month, or 1,350,000 GB. Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD... Let's suppose you even used the cheaper egress from Cloudfront rather than serving from S3 (lol @ your wallet if you serve 1 PB doing that). https://aws.amazon.com/blogs/aws/aws-data-transf…
Correct me if I'm wrong, but most colocation/dedicated server providers offer such prices. E.g. hetzner.com @ €1/TB, sprintdatacenter.pl @ €0.91/TB, dedicated.com @ $2/TB (or $600/month for an unmetered 1 Gbps connection).
But if you want S3/CDNs/, then yeah, they're expensive.
BTW per Cloudflare ToS[0]:
> Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#127Earlier quoted context omitted.
They're basically bragging they have something Apple really wants. Now they have a bunch of people at least interested in what they got. I'll say that's not a bad PR.
Also a great way to throw out massive red flags to any enterprise user that cares about privacy and non-disclosure. IP address data is pretty sensitive information, and throwing it out there like this, even in aggregate, is not OK because of what it shows. No matter how much PR this gets, this goes both ways.
Apple's ownership of 17.x is so well known it is described in an xkcd: https://xkcd.com/195/.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#128That's about $4000/month in bandwidth costs, assuming retail pricing. FYI he is bragging, not complaining. There are a dozen ways to reduce or eliminate this problem.
> That's about $4000/month in bandwidth costs You're an order of magnitude off. 45 TB per day is 1,350 TB in a month, or 1,350,000 GB. Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD... Let's suppose you even used the cheaper egress from Cloudfront rather than serving from S3 (lol @ your wallet if you serve 1 PB doing that). https://aws.amazon.com/blogs/aws/aws-data-transf…
Backblaze will send out data for 1 cent per GB, or $13,500. BunnyCDN apparently starts at 1 cent per GB in North America and Europe, and has a fewer-PoP plan that would be $5,550. And let's be fair, you don't need many points of presence for half-gig neural network models.
The other option is buying transit. Let's say we want the ability to deliver 80% of the 45TB in 4 hours as our peak rate. Then we need 20gbps of bandwidth. That's around $9,000 in a big US datacenter. Aim for a peak of 6/8 hours and it's $6,000/$4,500.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#129What does hugging face do? Do they implement models from papers and make them available for free?
Yes. I don’t know if that’s all that they do. They often port new Tensorflow models to PyTorch as well. They provide straightforward APIs, nice documentation, and clear tutorials. I use their stuff pretty regularly.