Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

81–90 of 422 posts

Re: An update on residential proxies and the scraper situation

#81

Post by Bratmon »

BoorishBears wrote: post #65
Bratmon wrote: post #59

Earlier quoted context omitted.

By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?

No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.

The article was called "The one we're commenting on"

Re: An update on residential proxies and the scraper situation

#83

Post by dylan604 »

colinsane wrote: post #34
tingletech wrote: post #3

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org . I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

If you scrape 6TB from across the web or grab it from one place, it's still 6TB

Re: An update on residential proxies and the scraper situation

#84

Post by fragmede »

dredmorbius wrote: post #64
fragmede wrote: post #63

Earlier quoted context omitted.

HN is exported to firebase, which you can hit directly, for that sort of purpose https://github.com/HackerNews/API

I know that. I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard. API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites. That said, a standard, say, JSON packaging of website contents available on request might be fun to have.

I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.

Re: An update on residential proxies and the scraper situation

#85

Post by m463 »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

At least anubis works for me. (I run umatrix)

Unfortunately whatever HN is using routinely blocks my login with "Sorry."

some websites just always give me 403.

Re: An update on residential proxies and the scraper situation

#86

Post by dylan604 »

andai wrote: post #41

>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses…

> do you want random IoT devices talking to your web server?

Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing

Re: An update on residential proxies and the scraper situation

#87

Post by khurs »

dang wrote: post #69
fragmede wrote: post #42

Earlier quoted context omitted.

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.

Ive been seeing a 'sorry' message occasionally when accessing older pages.

Is that a side effect of whatever you are doing?

Re: An update on residential proxies and the scraper situation

#88

Post by dylan604 »

fragmede wrote: post #44
eduction wrote: post #16

Can BitTorrent’s architecture contribute anything useful here? I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?

Re: An update on residential proxies and the scraper situation

#89

Post by binarymax »

colinsane wrote: post #34
tingletech wrote: post #3

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org . I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

Re: An update on residential proxies and the scraper situation

#90

Post by BoorishBears »

Bratmon wrote: post #81
BoorishBears wrote: post #65

Earlier quoted context omitted.

No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies. The article was called "The one we're commenting on"

> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.

Does that sound like your typical self-hosted blog?

Post reply on HN