Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

91–100 of 255 posts

Re: Let us serve you, but don't bring us down

#91

Post by crazydoggers »

crazydoggers wrote: post #18
TekMol wrote: post #3

What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?

Same way a library does. The information is made available to anyone who wants it, the copyrights are maintained intact, and they don’t attempt to profit from the material. Copyright law specifically allows for libraries and archives to make copies of copyrighted material. Without such laws, without libraries, knowledge could not be guaranteed to be shared freely among the public, resulting in ever growing knowledge…

I just wanted to add, while I don’t agree with the assessment that copyright holders need to be protected from something like archive.org, i don’t think the parent comment deserved to be flagged, so I vouched for it. I think the question was raised in sincerity, and I think it offers a point of discussion for those not familiar with the issues.

I don’t think it’s helpful to flag things people disagree with, as long as they don’t attempt to spread misinformation, or trolling etc. The parent phrased the topic as a question, meaning I believe they were open to understanding.

I also think it’s relative to the topic posted, as we’re talking about either an attack on archive.org, or a massive recopying of archive data by an unknown.

Talking to people with opposing views is important. Let’s not just shut people down if we don’t agree with something, especially when someone is asking a question.

Re: Let us serve you, but don't bring us down

#92

Post by Pmop »

pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

There's a demand, why not supply it, and make money while you are at it?

This reminds me of the RMT driven botting problem in WoW (World of Warcraft). Instead of fighting the neverending game of cat and mouse against botters, Blizzard just decided to supply the long reprimanded demand for in-game currency by creating the WoW token, and they make money while they're at.

Re: Let us serve you, but don't bring us down

#93

Post by schoen »

WA wrote: post #84
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

I usually make it a two step process: host example.com If this returns with an IP address, no need to talk to a registrar. Only if there is no IP address, I go with whois example.com

You could also do an in-between step of

  host -t soa example.com
which should give you domains that have any DNS record at all, not just an A record.

Re: Let us serve you, but don't bring us down

#94

Post by entropyie »

emporas wrote: post #61
8organicbits wrote: post #47

Earlier quoted context omitted.

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

Security is not all or nothing. There are many applications where adding a small bit of friction in the form of compute will stop 99.9% of abusive traffic. Visual captchas are a plague on the internet, but so is Blockchain mania.

I added FriendlyCaptcha to some of my sites, and stopped 100% of abusive traffic. Open source, user friendly, accessible to people with disabilities.

https://friendlycaptcha.com/

Most of us are not running amazon.com here.

Re: Let us serve you, but don't bring us down

#95

Post by RockRobotRock »

rapnie wrote: post #34
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

hCaptcha's accessibility model seems quite good to me: https://www.hcaptcha.com/accessibility

Re: Let us serve you, but don't bring us down

#96

Post by buro9 »

I'm waiting for Cloudflare to open source their web server as promised.

But failing that I'm likely to implement a web proxy that utilizes the wirefilter library that is already open sourced by Cloudflare (but no longer updated).

The tools to stop these attacks are reasonably trivial.

Most of the art of stopping them is observability.

When you can see every dimension of a TCP/UDP connection, the TLS handshake, the HTTP communication... Then the dimension by which an attack is being conducted is glaringly obvious.

Once you have the very obvious correlation, then you only need a blunt instrument of a tool that can block/deny/nullroute the attack.

What's hard isn't the block rules, What's hard is instrumenting enough observability to see the obvious correlation of an attack.

Even on my personal nginx a single request logs about 20 things. But there are literally hundreds of properties one can log if you have access to the entirety of the stack, and if you have access to log on it, then you have the ability to carry that context to a point in the code where you can block it.

Also, 10k qps really isn't a lot. So this should be treated as a warning sign, this attack was low volume even for amateur booter services.

Re: Let us serve you, but don't bring us down

#97

Post by Retr0id »

yyyk wrote: post #65

Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download. * If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have…

Last time I wanted to bulk-archive a bunch of urls, I asked about it, and sent a txt file full of URLs to someone and they put it in the archival queue.

Re: Let us serve you, but don't bring us down

#98

Post by emporas »

8organicbits wrote: post #87
emporas wrote: post #79

Earlier quoted context omitted.

I am talking about Bitcoin BSV. If the purpose of blockchain was solely to provide PoW, then there could exist a lot of blockchains, a thousand maybe. Economic incentives ensure that the most inefficient of blockchains will be put out of the market. So some of them, like btc will soon be put out of the market, because speed and microtransactions is the two factors every blockchain competes on. https://www.binance.com…

> This coin is not listed on Binance for trade and service. An obscure coin my users cannot purchase through major exchanges is a really poor solution. Sorry, that makes no sense.

I answered on the other comment, but at that point, in case there is a blockchain, which supports a millionth of a cent transaction, then exchanges (Binance, Coinbase etc) are not so useful. If every person just needs one cent for a million http requests, then one guy in your neighbourhood or your town, or your city might have some of it, you message him and he will send you a cent for free. You buy him a coffee, and he will give you in return, billions of http requests.

But again, economic incentives need to work their way into the system.

Re: Let us serve you, but don't bring us down

#99

Post by llui85 »

yyyk wrote: post #65

Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download. * If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have…

They have a Google Sheets "API" which I've used and works reasonably well:

https://archive.org/services/wayback-gsheets/

Re: Let us serve you, but don't bring us down

#100

Post by fsckboy »

RockRobotRock wrote: post #95
rapnie wrote: post #34

Earlier quoted context omitted.

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

hCaptcha's accessibility model seems quite good to me: https://www.hcaptcha.com/accessibility

proof of work sounds accessible to me
Post reply on HN