This is what success looks like if you charge money for a good or service.
Apple downloads ~45 TB of models per day from our S3 bucket
101–110 of 237 posts
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#102Earlier quoted context omitted.
Not with this amount of traffic it won't be free. 45 TB per day lol Cloudflare will be disabling your account and in contact for payment in a hurry. Go ahead and try, see how far their "free" tier really goes.
Just waiting for the Cloudflare CEO who lurks around to pop in here and offer it for free.
[deleted]
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#103Earlier quoted context omitted.
Not to mention that if Cloudflare CDN was in front of it this traffic would be free.
Not with this amount of traffic it won't be free. 45 TB per day lol Cloudflare will be disabling your account and in contact for payment in a hurry. Go ahead and try, see how far their "free" tier really goes.
For comparison, that's just about half the amount cdnjs delivers every day (on average) https://github.com/cdnjs/cf-stats/blob/master/2019/cdnjs_Aug...
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#104"Almost everyone" working on NLP uses one of hugginface's pretrained models at one point or another, sooner or later: https://github.com/huggingface/pytorch-transformers It's so damn convenient, and so nicely done. And they keep doing neat things like this one: https://github.com/huggingface/swift-coreml-transformers Kudos to Julien Chaumond et al for their work!
> Swift Core ML Why does he call them Swift Core ML? They are core ml, usable in swift and objective-c.
That repo is written in Swift, hence it is called Swift Core ML Transformers.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#105Earlier quoted context omitted.
I'm skeptical of the number of 500+ MB files the CloudFlare CDN would actually cache... Does anyone have any numbers on this?
Correct; Cloudflare doesn't cache large asset files (I think anything more than 2MB?) by default. It's not that kind of CDN... at least, not for free it's not. Of course, you can trick Cloudflare into caching your large media assets using some funky Page Rules... but I wouldn't suggest it. Mostly just for moral reasons. If you have that much traffic, you should be making some money off it and then paying Cloudflare w…
512mb: https://support.cloudflare.com/hc/en-us/articles/200172516-U...
And a "cache everything" page rule tends to cache literally any file type, but it's not a great idea to push media files through CF due to the TOS prohibiting "disproportionate amounts of non-web content".
2mb might be referencing the limit for Workers KV.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#106I don't have high hopes for his business prospects if this is how he handles one of the richest companies in the world clearly having a high need for something his company offers. Maybe spend less time on Twitter and more on your business model?
[deleted]
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#107Earlier quoted context omitted.
> That's about $4000/month in bandwidth costs You're an order of magnitude off. 45 TB per day is 1,350 TB in a month, or 1,350,000 GB. Show me somewhere you can get a petabyte of egress inside a calendar month for 4 figures USD... Let's suppose you even used the cheaper egress from Cloudfront rather than serving from S3 (lol @ your wallet if you serve 1 PB doing that). https://aws.amazon.com/blogs/aws/aws-data-transf…
How the hell are they paying that bill then??
ingress/egress is usually free within the region AFAIR
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#108So these jokers at apple publish a paper by using code from huggingface.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#109Hosting terabytes of data on an S3 bucket where people would download 45TB per month ($0.023/GB == $1000+/month) sounds like a really expensive way to distribute your data to people...
It is download 45 TB PER DAY. Should use the egress data price for the calculation instead of the hosting price.
Re: Apple downloads ~45 TB of models per day from our S3 bucket
#110In a few weeks he can just point Apple's IP range to a shared iCloud folder.
That'll get them them Apple's attention pretty quickly I'd assume.
Now that’d be a bit ironic to get a "your account is using excessive data bandwidth " and be able to respond with "incorrect, your ip’s are using too much bandwidth ".