Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

71–80 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#71

Post by draw_down »

hinkley wrote: post #45
spiderfarmer wrote: post #2

Was it a real FB crawler or a random buggy homemade bot masquerading as one? Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...

Someone up to no good using a false flag would not surprise me in the slightest. "who do people already like to hate? I'll pretend to be them."

Why is it so hard for people here to believe that Facebook could have a crawler misbehaving? It is all over this thread, even though multiple people are saying they saw the same thing and confirmed the IP belonged to FB.

What gives?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#72

Post by Wingy »

napolux wrote: post #57
hinkley wrote: post #45

Earlier quoted context omitted.

Someone up to no good using a false flag would not surprise me in the slightest. "who do people already like to hate? I'll pretend to be them."

Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#73

Post by napolux »

huac wrote: post #30
akerro wrote: post #25

Earlier quoted context omitted.

You can check list of IP addresses?

the OP did not (explicitly) check, but you can check if the IP falls into a range allocated to Facebook e.g. https://ipinfo.io/AS32934

Of course I've checked. Here is a recent example. https://news.ycombinator.com/item?id=23491455

Re: A Facebook crawler was making 7M requests per day to my stupid website

#77

Post by Sohcahtoa82 »

laurentdc wrote: post #8

Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent? I wonder if you can bypass paywalls or see things that aren't generally presented to regular users

You wouldn't even need an extension. Both browsers allow you to change your User-agent. An extension just makes it a little easier.

I have a UA switcher on Firefox so I can change my UA in two clicks. One of the built-in UA options is indeed the Googlebot.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#78

Post by detaro »

Wingy wrote: post #72
napolux wrote: post #57

Earlier quoted context omitted.

Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.

If they made actual requests that went through, a connection got established. That won't happen with a faked source.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#79

Post by yupyup54133 »

bberenberg wrote: post #61

Doesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.

When someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".

Re: A Facebook crawler was making 7M requests per day to my stupid website

#80

Post by ekimekim »

elviswolcott wrote: post #58
nitrogen wrote: post #18

Earlier quoted context omitted.

I like link previews. I don't like when they are managed server-side instead of client-side.

The issue with doing them client side (other than the missed cache opportunity of using a server) is that it means you're having every user in the chat send a request to an arbitrary URL from their device. Also CORS makes this impossible 99% of the time.

The third approach, which Signal uses as a privacy preservation measure, is for the sender to generate the link preview on send. Of course, this then opens up abuse where the sender can arbitrarily control the "link preview" to say whatever they want.
Post reply on HN