Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

31–40 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#31

Post by jon_richards »

dragonwriter wrote: post #19
toomuchtodo wrote: post #12

Earlier quoted context omitted.

What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.

Then the action needs to be harmless, because GET is defined as not merely idempotent but also safe. Didn't we already learn this lesson after all the unsafe-GET problems unveiled when prefetching browser accelerators came on the scene in, IIRC, the late 1990s?

Most of the time, GET should not affect system state at all (other than caching, etc). Idempotent can still have effects on the first access. Even deletion can be idempotent.

I generally see PUT and PATCH classified as idempotent (for the same input).

Re: A Facebook crawler was making 7M requests per day to my stupid website

#32

Post by dragonwriter »

thaumasiotes wrote: post #28
dragonwriter wrote: post #19

Earlier quoted context omitted.

Then the action needs to be harmless, because GET is defined as not merely idempotent but also safe. Didn't we already learn this lesson after all the unsafe-GET problems unveiled when prefetching browser accelerators came on the scene in, IIRC, the late 1990s?

> Then the action needs to be harmless to repeat, because GET is defined as idempotent. No, this is a terrible response. The action needs to be harmless to execute every time , not just every time after the first time. HTTP DELETE is conceptually idempotent, but you don't want to be deleting stuff with GET requests. That's why the standard provides a DELETE method! The distinction that really matters is safe/unsafe,…

> The action needs to be harmless to execute every time, not just every time after the first time.

You are correct, and the grandparent post has been updated appropriately.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#33

Post by _lqaf »

leafo wrote: post #9

Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834 The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off.

Of course, sample size of 1, etc. It could have been coincidental.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#35

Post by systemvoltage »

crazygringo wrote: post #20
systemvoltage wrote: post #13

Earlier quoted context omitted.

This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.

I disagree completely, I find it extremely useful. Most links are unreadable (e.g. an ID to a cloud file), and are truncated anyways even if readable. The preview gives me the title of the webpage, which is infinitely more useful -- especially letting me know whether it's a document I've already seen (and don't need to open) or something new, and if so, what.

To me it pollutes the discussion away from the messages and puts a bunch of thumbnails where it could just be a clean text conversation.

The thing that bothers me is also that I, as a user, have no choice - it just does it automatically.

Do you like IRC?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#36

Post by dragonwriter »

thaumasiotes wrote: post #24
thinkindie wrote: post #15

Earlier quoted context omitted.

that's your own problem because you are going against the HTTP protocol standard. GET should be idempotent.

Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately. The problem with unsafe GET is that it conflicts with reality, not that it conflicts with the standard.

> Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately.

That's only against the standard of you advertise HEAD as a supported method on the resource, which converts it from a good idea in some circumstances to a bad one, so if there is a good example to support your claim, that isn't it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#37

Post by dependenttypes »

crazygringo wrote: post #16
toomuchtodo wrote: post #12

Earlier quoted context omitted.

What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.

Then like literally every search engine and website crawler ever does, the action will be performed. Which is why it's bad practice to design your website using GET for actions. That's what POST is for. I mean, using GET for actions will break so many things -- browser prefetching, link previews, the list is endless. If you use GET for actions, just... yikes.

> browser prefetching, link previews

Both of them are pretty bad. I do not see GET actions breaking anything that is not trash.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#39

Post by mikece »

crazygringo wrote: post #27

If you can verify it's actually coming from Facebook's IP address (as many other comments here suggest), you should absolutely get in touch with them, though I'm not sure how. Perhaps their security team. That's a very serious bug that they'd be glad to catch -- if it's affecting you it's surely affecting others. Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it b…

From the sounds of it the difference between Facebook's actual crawler and malware could be hard to define or differentiate, given the circumstances.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#40

Post by president »

I just thought of a malicious idea. If I were Amazon or some other cloud provider, I could hammer my customers' web sites with tons of requests to make them pay more for resource usage (network bandwidth, s3 calls, misc per/unit usage etc). It would be hard to trace as well. Wonder if people are already doing that today.
Post reply on HN