Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

161–170 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#161

Post by PetahNZ »

ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

My understanding is unless he's distributing it (not just serving it from a website) he doesnt need to release his changes. That's what the AGPL is for.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#162

Post by 0x44 »

timy2shoes wrote: post #150
RhodesianHunter wrote: post #142

Earlier quoted context omitted.

Despite how it sounds, I ask this with zero judgment and pure curiosity. Why do you care?

Please read https://sfconservancy.org/copyleft-compliance/principles.htm... . My understanding is that if you do not enforce your copyright (or copyleft in this case) you can lose the copyright.

Whilst you can lose a trademark for not enforcing it, you cannot lose a copyright by not enforcing it (in the United States).

Re: A Facebook crawler was making 7M requests per day to my stupid website

#163

Post by ruuda »

ricardobeat wrote: post #144
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

They look similar at a glance (border-top + Calluna font), so he might have taken inspiration from yours, but doesn't seem to have used any of your assets - the styles are clearly different and based on the WP 'BlankSlate' theme. (my personal blog had a top border like that a decade ago, when styles on the body were a novelty :))

I checked the source and it does indeed mention https://wordpress.org/themes/blankslate/, but judging from the screenshot there, that theme is just really a blank theme with no style at all, and it was used to include a different stylesheet.

The real style is at https://coding.napolux.com/wp-content/themes/coding.napolux...., which looks like normalize.css followed by a Wordpress adaptation of my stylesheet.

The similarity is more than superficial, the footer headers match exactly.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#164

Post by bkanber »

thaumasiotes wrote: post #28
dragonwriter wrote: post #19

Earlier quoted context omitted.

Then the action needs to be harmless, because GET is defined as not merely idempotent but also safe. Didn't we already learn this lesson after all the unsafe-GET problems unveiled when prefetching browser accelerators came on the scene in, IIRC, the late 1990s?

> Then the action needs to be harmless to repeat, because GET is defined as idempotent. No, this is a terrible response. The action needs to be harmless to execute every time , not just every time after the first time. HTTP DELETE is conceptually idempotent, but you don't want to be deleting stuff with GET requests. That's why the standard provides a DELETE method! The distinction that really matters is safe/unsafe,…

> No, this is a terrible response.

As an aside, this kind of hyperbole really gets under my skin. It wasn't a terrible response. That statement is already technically correct: GET is idempotent, and the definition of idempotency is that it is harmless to repeat.

Your gripe is that OP didn't mention that GET is not only idempotent but must also be "safe"; i.e. that it should not alter the resource. OP got it 50% correct.

Does that omission make his comment a "terrible response"? No -- just incomplete.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#165

Post by johnwheeler »

ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

Next time, have some courtesy for the author and the rest of us by requesting via personal exchange over email instead of hijacking the thread and distracting from the conversation.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#166

Post by ricardobeat »

jrockway wrote: post #134
jokoon wrote: post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated re…

There is a cost to apply throttling, you still have to receive those requests and filter/block them, plus incoming bandwidth.

The post mentions setting `og:ttl` and replying with HTTP 429 - Too Many Requests, and both being ignored.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#167

Post by ruuda »

napolux wrote: post #151
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

Sure man no problem. The code for my theme is here BTW with credits To your original blog https://github.com/napolux/coding.napolux.com/

Thanks, I’m flattered to see it be used as inspiration :)

I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#168

Post by pera »

lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address.

By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#169

Post by napolux »

ruuda wrote: post #167
napolux wrote: post #151

Earlier quoted context omitted.

Sure man no problem. The code for my theme is here BTW with credits To your original blog https://github.com/napolux/coding.napolux.com/

Thanks, I’m flattered to see it be used as inspiration :) I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

Sure, no problem

Re: A Facebook crawler was making 7M requests per day to my stupid website

#170

Post by dgellow »

jrockway wrote: post #134
jokoon wrote: post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated re…

The author specified that the crawler ignores 429 status code. So they do have some rate limit.
Post reply on HN