> I own a little website I use for some SEO experiments [...] can generate thousands of different pages.
Do you have more details on this? One could read that as if the page is designed to attract crawlers.
A Facebook crawler was making 7M requests per day to my stupid website
341–350 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#342Earlier quoted context omitted.
> e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that? If I released it on Github, under any license whatever? I’d more or less be expecting that. If it was about the 4hr of work that went into my blog theme, I wouldn’t be bothered at all . But then, I wouldn’t re…
So, you are okay with people violating other peoples licenses and ignoring copyright. Gotcha.
You can simultaneously follow other people's licenses to the letter while also not caring if other people don't follow your's.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#343Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?
Sure man no problem. The code for my theme is here BTW with credits To your original blog https://github.com/napolux/coding.napolux.com/
Am inexperienced in the technicals of this, is "taking inspiration" from someone else's website grounds for a copyright or gpl violation?
When I've worked with designers it usually starts "pick a site that inspires you", where the deliverable has resemblance.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#344Earlier quoted context omitted.
IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.
If they made actual requests that went through, a connection got established. That won't happen with a faked source.
Could be the IP address OP is showing is actually from an X-Forwarded header or other proxy header (but not the actual source of the packet).
Re: A Facebook crawler was making 7M requests per day to my stupid website
#345Earlier quoted context omitted.
It's perfectly valid to criticize the original license choice. GPLv3 is a very restrictive license, especially for what is essentially a micro blog (though I dislike the license for most open source software anyway). Add on the original author going after a bit of CSS, not even the main effort of the project in question, and you've got my "petty" comment.
You didn't answer my question. That pretty much says all that needs to be said.
Not the parent but your question and the implicit accusation is way too overblown. The design here is so generic that it can easily be used by tons of site out there. It's just a few lines of CSS here and there and if I have a design like that and come across something similar I would just chalk it up to someone with similar taste. Going out of your way to demand attribution for it is the very definition of petty.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#346Earlier quoted context omitted.
I made my site open source so others who like it can take a look at how it works, or give it their own twist. Visitors of my site can see that they have that freedom, but visitors of an adaptation might not know if it’s not stated anywhere.
You're not wrong to request attribution. I don't know why you're getting so much flak.
Lots of people don't like the GPL (I love it myself, for reasons stated in the GP), and seeing it enforced in real time brings that debate up again.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#347Earlier quoted context omitted.
Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.
It may focus on user's rights, but it still requires technical expertise to exercise half of the 4 freedoms: #1 (inspection & modification) and #3 (distributing your modifications). Merely knowing to ask "can I see the source code" I would put into the "technical user" realm. Overwhelmingly, most people don't know or don't care. The other two freedoms #0 (freedom to run) and #2 (freedom to share) are readily obvious…
The users don't have to be technical to benefit from those freedoms though - there's a level of indirection involved. I might not have personally scrutinized every line of the Linux kernel, but knowing that there are tons of people in the world with the ability and motivation to do that inspires confidence.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#348That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).
Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated re…
Rate limiting websites is a lot of work. You have to figure out what kind of limits you should set, you have to figure out if that should be on an IP or a /24 or an ASN, if it's an ASN, you have to figure out how to turn an IP into an ASN.
If your website is pretty fast, you probably won't necessarily care about a lot of hits from one source, but if it's supposed to be a small website on inexpensive hosting, and all those hits add up to real transfer numbers, maybe it's an issue.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#349Earlier quoted context omitted.
Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.
Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA. Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things. Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miser…
Is there an official error message for that? Because filtering no UA would trip me up every time I use wget. Most of the time, if I'm casually using wget for something, I don't bother with a UA. If sites started rejecting that, I'd like to get a clear error message, so I would not go crazy trying to figure out what the problem was. If I got a clear message "send a UA" then I would probably started wrapping my wget requests in a short bash script with some extra stuff thrown in to keep everyone happy. But I'd have to know what it is that is needed to keep everyone happy.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#350Earlier quoted context omitted.
I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.
Something about this idea sits uncomfortably with me. I also just had an idea / thought experiment based on your idea. We think of net neutrality as being for carriers and ISPs, but you could see it applied to a publicly accessible DNS service too. These DNS service providers are just as much part of the core service of the Internet as anyone else. It’s not a huge leap to require that those who operate a publicly acc…
Principled neutrality is fine for acceptable use. There’s no moral quandary in closing the door to abusers.