Downloading a file regularly - how hard can it be?
adblockplus.org
Downloading a file regularly - how hard can it be?
1–10 of 44 posts
Re: Downloading a file regularly - how hard can it be?
#2> with the effect that people always download on the same weekday
What's so bad about that?
Re: Downloading a file regularly - how hard can it be?
#3A common solution to this problem, is to make a 2 stage process, where step 1 is a request of "should I download?", where there are 2 possible replies: "no, check again in N time" and "yes, here is a token". Step 2 is then presenting the token to the api point for download, and getting the file.
On the server side, you don't even need specific instance tracking, just a simple decision based on current resource usage, and a list of valid tokens (optionally, they can expire in some short time to avoid other thundering herd type issues). Say, you set a max number of file transfers, or bandwidth or whatever metric makes sense to you, and you simply reply based on that metric. Further, you can smooth out your load with a bit of intelligence on setting N.
Even better, you get a cool side-effect: since the check isn't so resource intensive, you can set the time between checks lower, and make the updates less regular.
Now that I think of it: it seems that this would be a nice nginx plugin, with a simple client side library to handle it for reference. Anyone want to collaborate on this over the weekend? Should be relatively straight-forward.
Re: Downloading a file regularly - how hard can it be?
#4> with the effect that people always download on the same weekday What's so bad about that?
Everybody downloading at once makes for slow servers. The author probably could pay much less in hosting costs if everybody downloaded the file along a uniform disturbance.
Re: Downloading a file regularly - how hard can it be?
#5> with the effect that people always download on the same weekday What's so bad about that?
Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.
Re: Downloading a file regularly - how hard can it be?
#6> with the effect that people always download on the same weekday What's so bad about that?
Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.
Isn't that something that nginx/varnish should easily be able to handle? It is just a static file download after all...
Re: Downloading a file regularly - how hard can it be?
#7Earlier quoted context omitted.
Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.
Isn't that something that nginx/varnish should easily be able to handle? It is just a static file download after all...
Even serving a static file is a burden on the server when you have millions of requests.
Re: Downloading a file regularly - how hard can it be?
#8I would just sign it, stick it on S3, and forget it. Did I miss why that wasn't considered?
Re: Downloading a file regularly - how hard can it be?
#9Earlier quoted context omitted.
Isn't that something that nginx/varnish should easily be able to handle? It is just a static file download after all...
Even serving a static file is a burden on the server when you have millions of requests.
What are the exact numbers?
A quote from: http://wiki.nginx.org/Main :
> I currently have Nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).
Re: Downloading a file regularly - how hard can it be?
#10I would just sign it, stick it on S3, and forget it. Did I miss why that wasn't considered?
While this workaround has merit, it doesn't actually solve the underlying problem. I guess even Amazon will eventually pick up the phone and ask you to stop sending them weekly bandwidth spikes when the figures involved get large enough (I've personally seen this with another well known PaaS provider).