Downloading a file regularly - how hard can it be? • Hacker News Board

Re: Downloading a file regularly - how hard can it be?

#41

Post by AntiRush » Tue, Apr 10, 2012, 6:56 AM UTC

icebraining wrote: post #37

skeletonjelly wrote: post #34

Earlier quoted context omitted.

I wonder if there's a market for Bittorrent over HTTP? Node.js, websockets...surely it's possible?

All of those are strictly client-to-server, not P2P. You could in theory proxy bittorrent over it, but you wouldn't gain anything over just serving the file from the server. You can probably write a true P2P client as a Firefox extension, since its API gives you very low level access (raw sockets, for example), but certainly not for e.g. Chrome.

WebRTC[1] seems to be the perfect platform for these sorts of things. It's in Chrome dev channel / Firefox Alpha right now.

[1] http://www.webrtc.org/

Re: Downloading a file regularly - how hard can it be?

#42

Post by pbhjpbhj » Tue, Apr 10, 2012, 10:38 AM UTC

drzaiusapelord wrote: post #27

moe wrote: post #15

Earlier quoted context omitted.

I guess even Amazon will eventually pick up the phone Why would they? You'd have to push dozens of GBit/s to even appear on their radar. The only time they'll call you is when they can't charge your CC anymore (a sustained 1 GBit/s will set you back $1000/day at their current rate).

Yep, they'll happily bill you. I imagine the budget adblock plus has is pretty small and, you know, can't be subsidized with ads.

Didnt they alter it to allow some ads through by default; or was that all reverted?

Re: Downloading a file regularly - how hard can it be?

#43

Post by sophacles » Tue, Apr 10, 2012, 1:26 PM UTC

masklinn wrote: post #39

sophacles wrote: post #30

Earlier quoted context omitted.

You still seem to be missing the point. Cache-Control as implemented commonly, and by your description, will instantly serve every request the new file as soon as a new file is available. It takes into account exactly one variable: file age. The algorithm I describe takes into account variables which affect current system loading, and returns a "no, try again later", even when the file is actually different, because…

> If no token is available and no if-modified headers are sent, reply with: > 503 + Retry-After N That's cool. There's still no reason for the second url and the 307, and you're still getting hit with requests so you're not avoiding the request load, only the download. You're smoothing out bandwidth, but not CPU & sockets.

This is sort of true. I don't know of a way to simply limit the number of incoming sockets without getting a lot of ISP level involvement or just outright rejecting connections. It does limit the number of long-lived sockets for file transfer. On static file serves, I am assuming the cpu has plenty of spare capacity for doing the algorithm, so I am not worried about that. Finally I am assuming the limiting factor is bandwidth here, so bandwidth smoothing the main goal.

Re: Downloading a file regularly - how hard can it be?

#44

Post by tantalor » Fri, Apr 13, 2012, 4:41 PM UTC

rmc wrote: post #5

tantalor wrote: post #2

> with the effect that people always download on the same weekday What's so bad about that?

Server load goes really high on that day, and if you get more popular, you'll need more servers and hence more money.

Maybe it wasn't clear, but in this case the load is still distributed evenly across all days. I always download on Monday, you always download on Tuesday. The author suggests that this is not desirable.

FTA,

The “Total” row looks very nice, it’s exactly what we would like to have. However, our download interval matches the number of days now, with the effect that people always download on the same weekday (and those currently downloading on a Monday will continue downloading on a Monday).