281–290 of 291 posts
Is this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?
Earlier quoted context omitted.
This was a notable example in that category: https://news.ycombinator.com/item?id=41058318 Also https://news.ycombinator.com/item?id=38766512
I just tried these with ChatGPT (4o) and it got both of them right. That's not to say that you won't be able to find something that still works but I think that particular hole is closing fast.
Earlier quoted context omitted.
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
What happens if you use the proper rate limiting status of 429? It includes a next retry time [1]. I'm curious what (probably small) fraction would respect it. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
Bad actors don’t care and annoying actors would make fun of you for it on twitter
Earlier quoted context omitted.
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
> This allows legit users to get what they need. Of course they could have just used the site directly.
It sucks!
Earlier quoted context omitted.
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
I've wanted to but wasn't sure how to keep track of individuals. What works for you? IP Addresses, cookies, something else?
There’s some stuff you can do, like creating risk scores (if a user changes ip and uses the same captcha token, increase score). Many vendors do that, as does my captcha provider.
Earlier quoted context omitted.
This would be terrible for accessibility for users using a screen reader.
... will result in a link ignored by screen readers.
Earlier quoted context omitted.
But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?
robots.txt is a somewhat useful tool for keeping search engines in line, because it's rather easy to prove that a search engine ignores robots.txt: when a noindex page shows up in SERPs. This evidence trail does not exist for AI crawlers.
Earlier quoted context omitted.
When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?
robots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically generated.