Someone purporting to be working in SEO and getting hit by 7M requests, but offers not real proof and cannot link the webpage either?
Am I just getting too suspicious?
Edit: Was trying to make a (bad) joke.
A Facebook crawler was making 7M requests per day to my stupid website
131–140 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#132Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"
My "404-pests" fail2ban-client status, which drops everything making *.php requests (This machine has never had PHP installed...): 2a03:2880:10ff:14::face:b00c 2a03:2880:10ff:21::face:b00c 2a03:2880:11ff:1a::face:b00c 2a03:2880:11ff:1f::face:b00c 2a03:2880:11ff:2::face:b00c 2a03:2880:12ff:10::face:b00c 2a03:2880:12ff:1::face:b00c 2a03:2880:12ff:9::face:b00c 2a03:2880:12ff:d::face:b00c 2a03:2880:13ff:3::face:b00c 2a03…
I think we're together in this, my friend
Re: A Facebook crawler was making 7M requests per day to my stupid website
#133I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.
I tried. It was ignored.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#134That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).
Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it?
(I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated requests, for things like "link unfurling" in things like Slack. I am guessing the crawler ignores robots.txt because it is doing the request on behalf of a human user, not to create some sort of index. Google is attempting to standardize this widely-understood convention: https://developers.google.com/search/reference/robots_txt)
Re: A Facebook crawler was making 7M requests per day to my stupid website
#135I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.
> HTTP 429 -> ignored
It's in the article
Re: A Facebook crawler was making 7M requests per day to my stupid website
#136Earlier quoted context omitted.
Really? I felt that what happened to Aaron was a tragic injustice on many levels (from the fact that he was charged at all to the number of charges they threw at him), but for some reason I had always thought that it was fairly well-known that that was his intent. I have no idea where I got that impression from, but I do recall reading several articles about him, as well as his blog around that time. It's possible I…
Criminal law is about proving things, not in-charachter assumptions.
Intent is the sticking point for the vast majority of prosecutions. That's why you have a jury, not Compute-o-Bot 9000.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#137Earlier quoted context omitted.
Super optimized with Facebook button on each post then. Not my definition of 'super optimized', but you know.
Super optimized in a SEO sense, not in a pleasing the HN crowd kind of sense I'd assume ;)
You're right, but for some reason the whole SEO thing just winds me up. It's my opinion that 'good' SEO makes sites worse for actual people to use.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#138Someone purporting to be working in SEO and getting hit by 7M requests, but offers not real proof and cannot link the webpage either? Am I just getting too suspicious? Edit: Was trying to make a (bad) joke.
Ask me anything. That specific website is a personal project of mine which I'd like not to disclose.
As you can see from my blog I'm not selling anything, I don't even have a banner on my blog, so where your suspect is coming from?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#139> And then they’ll probably ask you to pay the bill
Can anyone comment on whether there would be any legal basis for this?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#140Hi Napolux,
It looks like your site is using a theme based on my website (https://ruudvanasseldonk.com/, source at https://github.com/ruuda/blog). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?