Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

261–270 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#261

Post by wheresmycraisin »

lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#262

Post by thephyber »

the_jeremy wrote: post #128
thephyber wrote: post #96

Earlier quoted context omitted.

Establish contractual damages in a ToS for the site. Prove violation and offender. Take to court and collect damages. Converting the effort into cash is tough, but the strategy exists. Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says: > By continuing to access the Website, You acknowledge and agree that each email a…

Has that ever worked? I can't find any record of judgments one way or the other, on their website or elsewhere.

I know ProjectHoneyPot was pushing a $x billion litigation against a spammer. I don't remember if/how that was resolved.

This guy[1] apparently spent years just suing email spammers and occasionally winning.

[1] https://www.danhatesspam.com/index.html

Re: A Facebook crawler was making 7M requests per day to my stupid website

#263

Post by rapind »

detaro wrote: post #241
rapind wrote: post #231

Earlier quoted context omitted.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

That makes sense. I couldn't come up with a shady reason why they would do it to be honest, but I was curious.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#264

Post by rurp »

mulmen wrote: post #147
falcolas wrote: post #90

Earlier quoted context omitted.

Why is this the owner's problem? Someone at Facebook should be filing the bug, or better yet instrumenting their systems so that incidents like this issue a wake-up-an-engineer alert. Fuck this culture of "it's up to the victim of our fuckup to file a bug report with us".

Not just “file a bug report” but also compile a time series graph (lol) and then pray that Facebook triages it correctly, which they have no incentive to do.

And to make things even more ridiculous, you need to sign in with a facebook account to even file a bug there.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#266

Post by yjftsjthsd-h »

detaro wrote: post #241
rapind wrote: post #231

Earlier quoted context omitted.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything. IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#267

Post by onli »

dahfizz wrote: post #217
onli wrote: post #211

Earlier quoted context omitted.

You see the contradiction, right? If it's about the HTML and the CSS the user has that code directly by visiting the site. No further action would be needed. If it's about something that creates the HMTL and CSS, then OP has no requirement because of the GPL just from people visiting the site and accessing the output CSS/HTML. Because that's not distributing the code as defined by the GPL. The original author should…

> If it's about the HTML and the CSS the user has that code directly by visiting the site. No further action would be needed. Wrong. Transmitting modified GPL works requires more than just distributing the source: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions…

First, that is for Conveying Modified Source Versions. See the definition of conveying:

> To “convey” a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.

Generally the common position here is to use the AGPL if you want to cover regular network access. Probably all a bit murky because the GPL does not really fit well to things like this. But mostly it does not apply here, pretty specifically by definition.

Maybe you'd have a point regardless if the source was directly the HTML and CSS. And it seems like I was wrong with the modification notice (but I wasn't thinking about modifications in particular). But it's not HTML and CSS directly. Having looked at the source in question now the transmitted HTML and CSS is evidently not the source code, as both is produced by template files.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#268

Post by verdverm »

gerdesj wrote: post #250
pera wrote: post #168

Earlier quoted context omitted.

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

Nice idea, I like the thinking. I'll tuck that away for use later. PowerDNS has LUA built in amongst a few other things. My stack of projects to do is growing at a hell of a rate and I'm not popping them off the stack fast enough.

I know the feeling, it's one of the reasons I'm working on https://github.com/hofstadter-io/hof

Check out the code generation parts and modules, they are the most mature. We have HRDs (like CRDs in k8s for anything) and a scripting language between bash and Python coming out soon too.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#269

Post by xg15 »

detaro wrote: post #241
rapind wrote: post #231

Earlier quoted context omitted.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

Obligatory link: https://webaim.org/blog/user-agent-string-history/

Re: A Facebook crawler was making 7M requests per day to my stupid website

#270

Post by thaumasiotes »

bkanber wrote: post #164
thaumasiotes wrote: post #28

Earlier quoted context omitted.

> Then the action needs to be harmless to repeat, because GET is defined as idempotent. No, this is a terrible response. The action needs to be harmless to execute every time , not just every time after the first time. HTTP DELETE is conceptually idempotent, but you don't want to be deleting stuff with GET requests. That's why the standard provides a DELETE method! The distinction that really matters is safe/unsafe,…

> No, this is a terrible response. As an aside, this kind of hyperbole really gets under my skin. It wasn't a terrible response. That statement is already technically correct : GET is idempotent, and the definition of idempotency is that it is harmless to repeat. Your gripe is that OP didn't mention that GET is not only idempotent but must also be "safe" ; i.e. that it should not alter the resource. OP got it 50% cor…

Yes, it was a terrible response. Here are some examples of idempotent requests:

- Change the email address registered to my account from owner@gmail.com to new_owner@136.com .

- Instead of sending my direct deposit to account XXXX XXXX at Bank of America, from now on, send it to account YYYY YYYY at Wells Fargo.

- Delete my account.

- Drop the database.

None of these have any business being available to GET requests. Objecting to a misconfigured endpoint on the grounds that the functionality it implements is not idempotent implies that the lack of idempotence is what was wrong. That's a bad thing to do - anyone who takes your lesson to heart is still going to screw themselves over, because you gave them terrible advice. They may do it more than they otherwise would have, because you gave them advice that directly endorses really bad ideas. Idempotence or the lack thereof is beside the point.

Messing up on endpoint idempotence means you might hurt the feelings of a document. Messing up on endpoint safety means you might lose all your data as soon as anyone else links to your homepage. Or worse.

Post reply on HN