Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

111–120 of 255 posts

Re: Let us serve you, but don't bring us down

#111

Post by emporas »

sph wrote: post #107
emporas wrote: post #60

Earlier quoted context omitted.

In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…

Proof-of-work was originally introduced by Adam Back's hashcash, to fight email spam. Bitcoin is an evolution of that idea applied to digital cash. https://en.wikipedia.org//wiki/Hashcash

Yes, DDoS attacks are spam over the http protocol. Spam is spam over the imap protocol. Overwhelming a server with too many download requests, is not spam but it has the same effect. Calling the police every ten minutes because the door sounds like someone tries to break in, is spam.

Re: Let us serve you, but don't bring us down

#113

Post by shepherdjerred »

maksimur wrote: post #75
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

That's no worse than just relying on Arhcive.org to be sole provider

Re: Let us serve you, but don't bring us down

#114

Post by maksimur »

maksimur wrote: post #75
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

Thanks everyone for your reply. It made me reconsider my opinion. Though the downvotes are unnecessary. Use them when they're actually necessary. Not as gate keeping or as a "don't agree" button or whatever frivolous thing is passing through your mind.

Re: Let us serve you, but don't bring us down

#115

Post by RobotToaster »

entropyie wrote: post #94
emporas wrote: post #61

Earlier quoted context omitted.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

Security is not all or nothing. There are many applications where adding a small bit of friction in the form of compute will stop 99.9% of abusive traffic. Visual captchas are a plague on the internet, but so is Blockchain mania. I added FriendlyCaptcha to some of my sites, and stopped 100% of abusive traffic. Open source, user friendly, accessible to people with disabilities. https://friendlycaptcha.com/ Most of us…

Friendlycaptcha isn't open source, only the widget/client is. The sever seems to be closed source.

Re: Let us serve you, but don't bring us down

#116

Post by tannhaeuser »

TekMol wrote: post #3

What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?

A question indeed that has to be answered before everything else. We can't criticize eg search engines for "previewing" content on search result pages and for sending you to copycat sites with the most ads and ignore gross content copying at the same time. We also have these "helpful" insertions of links to archive.ph copies here when the original story is paywalled and sometimes even just ad-ridden - to copied content on archive.ph with third-party ads which I find at least unethical.

That said, personally I do use Wayback machine about once a month for genuine search of historic content.

Re: Let us serve you, but don't bring us down

#117

Post by RugnirViking »

emporas wrote: post #60
rapnie wrote: post #34

Earlier quoted context omitted.

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…

I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)

Re: Let us serve you, but don't bring us down

#118

Post by RobotToaster »

rapnie wrote: post #34
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

There's also (the imaginatively named) powcaptcha https://git.sequentialread.com/forest/pow-captcha

Re: Let us serve you, but don't bring us down

#119

Post by bazmattaz »

tastysandwich wrote: post #33

I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.

Yes, this is the way. When I was purchasing a car last year I scraped a popular used car website for cars I liked to keep track of deals. I added a random sleep in between each page so the entire script took a few hours to run.

Re: Let us serve you, but don't bring us down

#120

Post by marginalia_nu »

alentred wrote: post #103

Just yesterday there was a comment here on HN [1] about https://jsonip.com , which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417

Yeah I'm getting this shit too with Marginalia Search. I'm getting about 2-2.5M queries per day that are definitely from bots that would 100% sink my server if they somehow go through the bot mitigation. It peaks at hundreds of search queries per second. To be clear these are search queries, and search queries typically trigger disk reads of about ~10-20 Mb.

I get about 20,000 queries per day that may be human.

Post reply on HN