Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Downloading a file regularly - how hard can it be?

adblockplus.org

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Downloading a file regularly - how hard can it be?

#31

Post by nitrox »

kogir wrote: post #8

I would just sign it, stick it on S3, and forget it. Did I miss why that wasn't considered?

It is too expensive. 1TB of bandwidth costs about $120. A project like adblock plus will be consuming about 3 - 4 TB a month which will add up to around $450 a month.

Adblock list subscriptions are maintained and hosted by individual people who do at their spare time. They mostly pay for the servers out of their pockets. As one of the co-author of popular adblock list, I wouldn't want to break my bank to pay for S3 hosting. Our current solutions works out and when we reach our bandwidth limit, we could just simply buy addition TB of bandwidth at a much cheaper price than S3.

Btw, i just made a rough calculation using AWS simple monthly calculator. So correct me if I am wrong about S3 pricing.

Re: Downloading a file regularly - how hard can it be?

#32

Post by nitrox »

K2h wrote: post #18

call me oldschool, but having a huge peak demand is the perfect application for distributed source, like torrent. I know it is much more complicated to introduce P2P and way more risky if it gets poisoned, but it seems to me this underlying problem of huge peak demand was solved 10 years ago.

but there is a problem with bittorrent. Most Schools and works places block bittorrent. We would need to fallback to http or any other method that works in restricted places.

Re: Downloading a file regularly - how hard can it be?

#33

Post by j_s »

onli wrote: post #16
rb2k_ wrote: post #9

Earlier quoted context omitted.

What are the exact numbers? A quote from: http://wiki.nginx.org/Main : > I currently have Nginx doing reverse proxy of over tens of millions of HTTP requests per day (thats a few hundred per second) on a single server. At peak load it uses about 15MB RAM and 10% CPU on my particular configuration (FreeBSD 6).

https://easylist.adblockplus.org/blog/2011/09/01/easylist-st... is the first thing i found, talking about 11.5 million total users and 80% of them using easylist, 9.2 million. According to the blogpost, the still existing (i just noticed that this was after the update-behaviour change) monday-peak was 118.5% of the expected (week total / 7) - 73 million download in august, so (73 / 4 / 7) * 1.185 = 3 million for that…

+ SSL

Re: Downloading a file regularly - how hard can it be?

#34

Post by skeletonjelly »

nitrox wrote: post #32
K2h wrote: post #18

call me oldschool, but having a huge peak demand is the perfect application for distributed source, like torrent. I know it is much more complicated to introduce P2P and way more risky if it gets poisoned, but it seems to me this underlying problem of huge peak demand was solved 10 years ago.

but there is a problem with bittorrent. Most Schools and works places block bittorrent. We would need to fallback to http or any other method that works in restricted places.

I wonder if there's a market for Bittorrent over HTTP? Node.js, websockets...surely it's possible?

Re: Downloading a file regularly - how hard can it be?

#35

Post by tedunangst »

nitrox wrote: post #31
kogir wrote: post #8

I would just sign it, stick it on S3, and forget it. Did I miss why that wasn't considered?

It is too expensive. 1TB of bandwidth costs about $120. A project like adblock plus will be consuming about 3 - 4 TB a month which will add up to around $450 a month. Adblock list subscriptions are maintained and hosted by individual people who do at their spare time. They mostly pay for the servers out of their pockets. As one of the co-author of popular adblock list, I wouldn't want to break my bank to pay for S3 h…

Terabytes per month? That's insane. That's a million users (I can believe) downloading a megabyte (I can't quite believe). It appears my patterns.ini file is 600K, or about 150K compressed, so if I download it 30/5 = 6 times a month, that's... a megabyte. Wow.

Re: Downloading a file regularly - how hard can it be?

#36

Post by pjscott »

moe wrote: post #11

I assume changes are usually small, you may want to try serving diffs? I.e. have the clients poll for the md5 of their _current_ list-version. On the server store the diff that will upgrade them to the current version under that filename. If a client requests an unknown md5 (e.g. because he has no list or his list is corrupted) default him to a patch that contains the full file. This requires a little logic on both e…

This is probably the Right Way, but it would be more work than minor tweaks to the delay logic.

Re: Downloading a file regularly - how hard can it be?

#37

Post by icebraining »

skeletonjelly wrote: post #34
nitrox wrote: post #32

Earlier quoted context omitted.

but there is a problem with bittorrent. Most Schools and works places block bittorrent. We would need to fallback to http or any other method that works in restricted places.

I wonder if there's a market for Bittorrent over HTTP? Node.js, websockets...surely it's possible?

All of those are strictly client-to-server, not P2P. You could in theory proxy bittorrent over it, but you wouldn't gain anything over just serving the file from the server.

You can probably write a true P2P client as a Firefox extension, since its API gives you very low level access (raw sockets, for example), but certainly not for e.g. Chrome.

Re: Downloading a file regularly - how hard can it be?

#38

Post by mhaymo »

sliverstorm wrote: post #20
sophacles wrote: post #3

A common solution to this problem, is to make a 2 stage process, where step 1 is a request of "should I download?", where there are 2 possible replies: "no, check again in N time" and "yes, here is a token". Step 2 is then presenting the token to the api point for download, and getting the file. On the server side, you don't even need specific instance tracking, just a simple decision based on current resource usage,…

This is the more robust solution. The simple solution would be to generate a random number and convert that to a time of the week :)

But if you define a week as 7 days, then you will still experience a Monday peak for work computers. It doesn't solve the problem at all.

Re: Downloading a file regularly - how hard can it be?

#39

Post by masklinn »

sophacles wrote: post #30
masklinn wrote: post #29

Earlier quoted context omitted.

> The solution I was suggesting is similar to what you are talking about, but also has the feature of smoothing the load curves. It has no more feature of smoothing the load curve than using Cache-Control with the right max-age. > My algorithm is that logic It is no more that logic than doing what I outlined with proprietary behaviors. > this allows better control should the client ignore the etags by making the whol…

You still seem to be missing the point. Cache-Control as implemented commonly, and by your description, will instantly serve every request the new file as soon as a new file is available. It takes into account exactly one variable: file age. The algorithm I describe takes into account variables which affect current system loading, and returns a "no, try again later", even when the file is actually different, because…

> If no token is available and no if-modified headers are sent, reply with:

> 503 + Retry-After N

That's cool. There's still no reason for the second url and the 307, and you're still getting hit with requests so you're not avoiding the request load, only the download. You're smoothing out bandwidth, but not CPU & sockets.

Re: Downloading a file regularly - how hard can it be?

#40

Post by tripzilch »

tedunangst wrote: post #35
nitrox wrote: post #31

Earlier quoted context omitted.

It is too expensive. 1TB of bandwidth costs about $120. A project like adblock plus will be consuming about 3 - 4 TB a month which will add up to around $450 a month. Adblock list subscriptions are maintained and hosted by individual people who do at their spare time. They mostly pay for the servers out of their pockets. As one of the co-author of popular adblock list, I wouldn't want to break my bank to pay for S3 h…

Terabytes per month? That's insane. That's a million users (I can believe) downloading a megabyte (I can't quite believe). It appears my patterns.ini file is 600K, or about 150K compressed, so if I download it 30/5 = 6 times a month, that's... a megabyte. Wow.

Wow, that suggestion elsewhere in the thread, to serve diffs instead seems rather important now :)
Post reply on HN