231–240 of 288 posts
Earlier quoted context omitted.
*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
what do people use to get keyword alerts in HN?
I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.
Interesting approach. The scraper-vs-site-owner arms race is real. On the flip side of this discussion - if you're building a scraper yourself, there are ways to be less annoying: 1. Run locally instead of from cloud servers. Most aggressive blocking targets VPS IPs. A desktop app using the user's home IP looks like normal browsing. 2. Respect rate limits and add delays. Obvious but often ignored. 3. Use RSS feeds wh…
Without wanting to upset anyone - what makes you interested in sharing tips for team scraper? (Overgeneralising a bit) site owners are mostly cting for public benefit whereas scrapers act for their own benefit/for private interests. I imagine most people would land on team site-owner, if they were asked. I certainly would. P.S. is the best way to scrape fairly just to respect robots.txt?
This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can…
If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…
Earlier quoted context omitted.
If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…
or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff
I still don’t understand why a rate-limiting approach is not preferred. Why should I care if the abuse is coming from a bot or the world’s fastest human? Is there a “if you need to rate limit you’ve already lost” issue I’m not thinking of?
> checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Serving different contents to search engines is called "cloaking" and can get you banned from their indexes.
Which for better or worse is a large portion of the modern internet.