Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

181–190 of 422 posts

Re: An update on residential proxies and the scraper situation

#181

Post by ValdikSS »

noncoml wrote: post #127
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Or you can go full Reddit and just block anything that seems even remotely suspicious. Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too. Using VPN? Blocked. Your iPhone is too old, blocked. Your screen brightness too low? Believe or not, blocked.

My forum got scraped so hard that the ISP blackholed the IPv4 several times a week.

I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.

Re: An update on residential proxies and the scraper situation

#182

Post by miyuru »

Groxx wrote: post #79
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

For me Cloudflare is worse, it takes more than 5 seconds, where as anubius take 1-2 secs.

funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)

and most of the time, its on marketing product pages like in framework main site, which can be cached.

Re: An update on residential proxies and the scraper situation

#183

Post by geocar »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

> but PoW scales

Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.

> becoming much more obvious and easy to block, or they have to use massive amounts of compute.

If you believe this, please contact me: I think compute is free[1] and can probably help you out.

[1]: https://news.ycombinator.com/item?id=30175269

Re: An update on residential proxies and the scraper situation

#184

Post by charcircuit »

ccgreg wrote: post #175
charcircuit wrote: post #113

Earlier quoted context omitted.

But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.

We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.

Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:

1. When the page itself was last updated

2. When the crawled copy was last updated

In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.

The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.

Re: An update on residential proxies and the scraper situation

#185

Post by klamann »

I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.

The FSF has the right idea about all this:

> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.

> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.

https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...

Re: An update on residential proxies and the scraper situation

#186

Post by procaryote »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour

Millions per hour is tens per second though; perhaps the fix is performance improvements

Re: An update on residential proxies and the scraper situation

#187

Post by MikeRichardson »

noncoml wrote: post #127
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Or you can go full Reddit and just block anything that seems even remotely suspicious. Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too. Using VPN? Blocked. Your iPhone is too old, blocked. Your screen brightness too low? Believe or not, blocked.

Parks and Rec reference?

Re: An update on residential proxies and the scraper situation

#188

Post by swinglock »

geocar wrote: post #183
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

> but PoW scales Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest. > becoming much more obvious and easy to block, or they have to use massive amounts of compute. If you believe this, please contact me: I think compute is free[1] and can probably help you out. [1]: https://news.ycombinator.co…

Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.

Re: An update on residential proxies and the scraper situation

#189

Post by iiiwio »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

[flagged]

Re: An update on residential proxies and the scraper situation

#190

Post by jazzyjackson »

alightsoul wrote: post #106
Groxx wrote: post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
Post reply on HN