Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Downloading a file regularly - how hard can it be?

adblockplus.org

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Downloading a file regularly - how hard can it be?

#11

Post by moe »

I assume changes are usually small, you may want to try serving diffs?

I.e. have the clients poll for the md5 of their _current_ list-version.

On the server store the diff that will upgrade them to the current version under that filename. If a client requests an unknown md5 (e.g. because he has no list or his list is corrupted) default him to a patch that contains the full file.

This requires a little logic on both ends (diff/patch), but would probably slash your bandwidth requirements to a fraction.

A little napkin math:

25 lists * 150kb * 1mio fetches = ~3.75T

vs

25 lists * 1kb (patch) * 1mio fetches = 25G (0.025T)

Re: Downloading a file regularly - how hard can it be?

#12

Post by [deleted] »

rb2k_ wrote: post #9
onli wrote: post #7

Earlier quoted context omitted.

Even serving a static file is a burden on the server when you have millions of requests.

What are the exact numbers? A quote from: http://wiki.nginx.org/Main : > I currently have Nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

[deleted]

Re: Downloading a file regularly - how hard can it be?

#13

Post by ComputerGuru »

rb2k_ wrote: post #6
rmc wrote: post #5

Earlier quoted context omitted.

Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.

Isn't that something that nginx/varnish should easily be able to handle? It is just a static file download after all...

CPU and bandwidth are entirely different issues. Sure, nginx can handle the processing. But do you have the piping to match?

A run of the mill dedicated server has a 100mbit uplink. Do the math. (Hint: it's easy to saturate in no time).

Re: Downloading a file regularly - how hard can it be?

#14

Post by prostoalex »

ComputerGuru wrote: post #13
rb2k_ wrote: post #6

Earlier quoted context omitted.

Isn't that something that nginx/varnish should easily be able to handle? It is just a static file download after all...

CPU and bandwidth are entirely different issues. Sure, nginx can handle the processing. But do you have the piping to match? A run of the mill dedicated server has a 100mbit uplink. Do the math. (Hint: it's easy to saturate in no time).

Has anybody tried https://developers.google.com/speed/pagespeed/service for this?

Re: Downloading a file regularly - how hard can it be?

#15

Post by moe »

forgotusername wrote: post #10
kogir wrote: post #8

I would just sign it, stick it on S3, and forget it. Did I miss why that wasn't considered?

While this workaround has merit, it doesn't actually solve the underlying problem. I guess even Amazon will eventually pick up the phone and ask you to stop sending them weekly bandwidth spikes when the figures involved get large enough (I've personally seen this with another well known PaaS provider).

I guess even Amazon will eventually pick up the phone

Why would they?

You'd have to push dozens of GBit/s to even appear on their radar. The only time they'll call you is when they can't charge your CC anymore (a sustained 1 GBit/s will set you back $1000/day at their current rate).

Re: Downloading a file regularly - how hard can it be?

#16

Post by onli »

rb2k_ wrote: post #9
onli wrote: post #7

Earlier quoted context omitted.

Even serving a static file is a burden on the server when you have millions of requests.

What are the exact numbers? A quote from: http://wiki.nginx.org/Main : > I currently have Nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

https://easylist.adblockplus.org/blog/2011/09/01/easylist-st... is the first thing i found, talking about 11.5 million total users and 80% of them using easylist, 9.2 million. According to the blogpost, the still existing (i just noticed that this was after the update-behaviour change) monday-peak was 118.5% of the expected (week total / 7) - 73 million download in august, so (73 / 4 / 7) * 1.185 = 3 million for that specific list and 3.75 for all of them. Hope i didn't miscalculate ^^ Add to that the growth till than. The easylist.txt seems to have a size of 528kb.

Re: Downloading a file regularly - how hard can it be?

#17

Post by kijin »

rmc wrote: post #5
tantalor wrote: post #2

> with the effect that people always download on the same weekday What's so bad about that?

Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.

Use a CDN? It's not like the list is tailored to anybody.

Re: Downloading a file regularly - how hard can it be?

#18

Post by K2h »

call me oldschool, but having a huge peak demand is the perfect application for distributed source, like torrent. I know it is much more complicated to introduce P2P and way more risky if it gets poisoned, but it seems to me this underlying problem of huge peak demand was solved 10 years ago.

Re: Downloading a file regularly - how hard can it be?

#19

Post by oconnor0 »

prostoalex wrote: post #14
ComputerGuru wrote: post #13

Earlier quoted context omitted.

CPU and bandwidth are entirely different issues. Sure, nginx can handle the processing. But do you have the piping to match? A run of the mill dedicated server has a 100mbit uplink. Do the math. (Hint: it's easy to saturate in no time).

Has anybody tried https://developers.google.com/speed/pagespeed/service for this?

This is just downloading a single static text file so there's nothing to optimize.

Re: Downloading a file regularly - how hard can it be?

#20

Post by sliverstorm »

sophacles wrote: post #3

A common solution to this problem, is to make a 2 stage process, where step 1 is a request of "should I download?", where there are 2 possible replies: "no, check again in N time" and "yes, here is a token". Step 2 is then presenting the token to the api point for download, and getting the file. On the server side, you don't even need specific instance tracking, just a simple decision based on current resource usage,…

This is the more robust solution. The simple solution would be to generate a random number and convert that to a time of the week :)
Post reply on HN