Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

71–80 of 255 posts

Re: Let us serve you, but don't bring us down

#71

Post by phkx »

p-e-w wrote: post #55
akiselev wrote: post #25

Earlier quoted context omitted.

Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by…

Rate limiting is (usually) done on a per-host basis. The only users who would be adversely affected are those who perform hundreds of requests per second from a single system. Virtually every major website has per-host inbound request limits. This is completely standard practice and Archive.org is the odd one here. And DDoS isn't the only concern. Legitimate users that run poorly written Python scripts that make insa…

It‘s rather per public IP, such that e.g. behind a corporate proxy you may experience rate limiting even for regular use, just because you’re sharing a few IP addresses with a large number of users. Better hope that you get an increased quota for your external IPs in that case.

Re: Let us serve you, but don't bring us down

#72

Post by vaylian »

pierat wrote: post #7
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

QPS seems to be "queries per second". I first thought it was something like "quadrillion bytes per second" or some newer over-the-top data measurement :)

I haven't heard of that abbreviation before. But there is a stub on Wikipedia if anyone cares about improving that article: https://en.wikipedia.org/wiki/Queries_per_second

Re: Let us serve you, but don't bring us down

#73

Post by brylie »

Roark66 wrote: post #67

I just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value. How about they actually fund something really worthy of preservation? Of course it is archive.org role t…

The Internet Archive lists some of their major donors on the About IA page:

https://archive.org/about/

Since they are a U.S. 501(c)(3), they also publish annual reports, which can be downloaded e.g., at the ProPublica Nonprofit Explorer

https://projects.propublica.org/nonprofits/organizations/943...

Re: Let us serve you, but don't bring us down

#74

Post by chaxor »

dan_wood wrote: post #46
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

[flagged]

Torrents are the backbone of a lot of academic research downloads. They're the absolute best way to distribute neural network weights, or large models used in physical science studies. That or IPFS. If you use huggingface or dockerhub for anything like this, seriously consider switching, as it will be extremely useful when those centralized systems inevitably fail.

Re: Let us serve you, but don't bring us down

#75

Post by maksimur »

x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

Re: Let us serve you, but don't bring us down

#76

Post by elcomet »

emporas wrote: post #69
comex wrote: post #63

Earlier quoted context omitted.

> Blockchain economic incentives ensure that ASICs will be used by the miners first and foremost. So instead, what, you want people to buy BTC and send small amounts of it to websites? Problem is, given varying income levels across society and across the world, one person's expensive micropayment is another person's almost free micropayment.

In a comment down below, i made it clear i was not referring to BTC. In the internet there is no centralized registry of naming projects. There are currently three projects in which the communities call themselves bitcoin. Any one of the other two bitcoins support much smaller microtransactions than BTC. Very true, that micropayments could vary on their relative cheapness across the global population. Let's put a num…

So maybe you should link to the one you are referring to, if there are three projects with the same name.

Re: Let us serve you, but don't bring us down

#78

Post by 5e92cb50239222b »

maksimur wrote: post #75
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

How is that worse than serving files via HTTP? With HTTP archive.org is the only peer that can serve files to you, with BitTorrent it will be one of the (hopefully) many peers, and will degrade to the same level of redundancy as HTTP if all other peers leave.

BitTorrent also supports web seeds and they don't even really have to keep a full client running, just embed an HTTP link into the .torrent file.

Re: Let us serve you, but don't bring us down

#79

Post by emporas »

elcomet wrote: post #76
emporas wrote: post #69

Earlier quoted context omitted.

In a comment down below, i made it clear i was not referring to BTC. In the internet there is no centralized registry of naming projects. There are currently three projects in which the communities call themselves bitcoin. Any one of the other two bitcoins support much smaller microtransactions than BTC. Very true, that micropayments could vary on their relative cheapness across the global population. Let's put a num…

So maybe you should link to the one you are referring to, if there are three projects with the same name.

I am talking about Bitcoin BSV. If the purpose of blockchain was solely to provide PoW, then there could exist a lot of blockchains, a thousand maybe. Economic incentives ensure that the most inefficient of blockchains will be put out of the market. So some of them, like btc will soon be put out of the market, because speed and microtransactions is the two factors every blockchain competes on.

https://www.binance.com/en/price/bitcoin-sv

Re: Let us serve you, but don't bring us down

#80

Post by tkgally »

brylie wrote: post #73
Roark66 wrote: post #67

I just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value. How about they actually fund something really worthy of preservation? Of course it is archive.org role t…

The Internet Archive lists some of their major donors on the About IA page: https://archive.org/about/ Since they are a U.S. 501(c)(3), they also publish annual reports, which can be downloaded e.g., at the ProPublica Nonprofit Explorer https://projects.propublica.org/nonprofits/organizations/943...

The Internet Archive has many smaller donors, too. I use the Archive frequently for a variety of purposes and derive immense value from it, so I donate to it every year.
Post reply on HN