Earlier quoted context omitted.
You could also publish a list of IP addresses.
As the article describes, it doesn't help, because the traffic originates from millions of unique residential IPs across hundreds of ASNs and countries.
So?
Creepy Crawlies
301–310 of 702 posts
Re: Creepy Crawlies
#302Earlier quoted context omitted.
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
If you block Brazil, they'll find an alternative, maybe then you can sue them.
Re: Creepy Crawlies
#303Earlier quoted context omitted.
I’m not a participant in this race. Are the AI companies worried about anything but their valuations? I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large. Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need…
[to prove you are not a crawler and read this comment, please deposit $2.00]
I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam.
Anyway, what’s wrong with making 2 cents? :)
Re: Creepy Crawlies
#304Earlier quoted context omitted.
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scraper…
> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers. There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
[flagged]
Re: Creepy Crawlies
#305[dead]
Re: Creepy Crawlies
#306At this point they are using residential proxies and stuff, and increasing the difficulty level is not going to help, among other things because they don't pay for it.
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
Re: Creepy Crawlies
#307Earlier quoted context omitted.
> I have an admin dashboard with a little leaderboard for which ones get the most stuffed You can't tease us like that without showing screenshots.
Haha here's a few snippets (will auto expire in a week, future readers): https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
(FYI, these are comically distorted on mobile.)
Re: Creepy Crawlies
#308Earlier quoted context omitted.
Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need . Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world probl…
Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together.
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
Re: Creepy Crawlies
#309Earlier quoted context omitted.
What would you suggest as an alternative? Block the bots yourself ?
Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.
I have tried cookie based sessions and I have tried multiple things. I have been battling DDOS and script kiddies longer than CF has been a thing. You’re wrong in that you think I could continue to maintain the site without CF in front of it, I would just shut it down. It’s not a profitable forum, it’s just a hobby.
Re: Creepy Crawlies
#310I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely? > Us PHP developers I can tell.
it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.