Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

281–290 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#281

Post by quickthrower2 »

stiray wrote: post #216
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I dont really understand what is the issue. On my welcome page (while all other urls are impossible to guess) i give browser something that requires a few seconds of cpu at 100% to crunch. And tracking some user action in between, visting tarpitted urls etc. In last few years no bot came through. Why bother with robots.txt, just give them something to break their teeths... (I would give you the url, but I just dont w…

How much have you mined?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#282

Post by quickthrower2 »

jokoon wrote: post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

In the ToS, add a charge of 2 guineas, 5 shilling, a sixpence and a peppercorn per request.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#283

Post by icebraining »

Nextgrid wrote: post #181
mikece wrote: post #56

Earlier quoted context omitted.

A funny story (well, funny because it didn't happen to me) was chronicled on a recent episode of Corey Quinn's "Whiteboard Confessions" -- https://www.lastweekinaws.com/podcast/aws-morning-brief/whit... -- where Slackbot's auto URL unfurling feature "clicked" an SNS alert unsubscribe link when a tech posted the a report including that link into a slack channel. If nobody lost their job over this triggering a SEV-1 al…

As far as I know the iOS implementation mitigates that by generating the preview on the sender side. The receiver side will not make a request to the URL unless they manually click on it.

Same for Whatsapp and Wire.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#284

Post by gerdesj »

verdverm wrote: post #268
gerdesj wrote: post #250

Earlier quoted context omitted.

Nice idea, I like the thinking. I'll tuck that away for use later. PowerDNS has LUA built in amongst a few other things. My stack of projects to do is growing at a hell of a rate and I'm not popping them off the stack fast enough.

I know the feeling, it's one of the reasons I'm working on https://github.com/hofstadter-io/hof Check out the code generation parts and modules, they are the most mature. We have HRDs (like CRDs in k8s for anything) and a scripting language between bash and Python coming out soon too.

I've tried to work out what your project does but I'm none the wiser. GEB is prominent on my bookshelf. I'm a syadmin and I got as far as "hollow wold" in Go or was it "Hail Marrow"? Can't remember.

I've checked out your repo for a look over tomorrow when I'm cough sober!

Re: A Facebook crawler was making 7M requests per day to my stupid website

#285

Post by nraynaud »

We had that with google in the past. We put a kml database on a web server and made the google maps API read them. Google went to get them without any throttling (my guess is that they primed their edge servers straight from us), the server went down so fast we sought there had been a hardware failure. We ended up putting a cdn just for google servers.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#286

Post by golem14 »

golem14 wrote: post #276
djxfade wrote: post #207

Earlier quoted context omitted.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

That was the idea. There's a name for it - zip bomb. https://en.wikipedia.org/wiki/Zip_bomb A superficial search leads to things like https://www.rapid7.com/db/modules/auxiliary/dos/http/gzip_bo... https://stackoverflow.com/questions/1459673 You really want to be careful about potentially breaking laws ...

Here is another article: https://www.blackhat.com/docs/us-16/materials/us-16-Marie-I-...

If FB supports brotli, a much bigger compression factor than 1000 is possible, apparently.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#287

Post by verdverm »

gerdesj wrote: post #284
verdverm wrote: post #268

Earlier quoted context omitted.

I know the feeling, it's one of the reasons I'm working on https://github.com/hofstadter-io/hof Check out the code generation parts and modules, they are the most mature. We have HRDs (like CRDs in k8s for anything) and a scripting language between bash and Python coming out soon too.

I've tried to work out what your project does but I'm none the wiser. GEB is prominent on my bookshelf. I'm a syadmin and I got as far as "hollow wold" in Go or was it "Hail Marrow"? Can't remember. I've checked out your repo for a look over tomorrow when I'm cough sober!

Stop by gitter and I'd be happy to explain more

Re: A Facebook crawler was making 7M requests per day to my stupid website

#288

Post by FridgeSeal »

quickthrower2 wrote: post #282
jokoon wrote: post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

In the ToS, add a charge of 2 guineas, 5 shilling, a sixpence and a peppercorn per request.

Nah make it a strand of saffron per request and get your moneys' worth out of them.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#289

Post by Shared404 »

johnwheeler wrote: post #225
Dahoon wrote: post #202

Earlier quoted context omitted.

> ... and the rest of us ... Please don't try to police the thread and speak for yourself only.

“And the rest of us, excluding Dahoon”

You can add me to that list.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#290

Post by userbinator »

maxk42 wrote: post #195
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

GPL license states you must provide source code to the end-user. In this case, the end-user is legally Napolux, not the visitors to his site.

In any case, View -> Source

The source code of a public website is automatically provided to everyone who visits it. This fact is unfortunately not well-known.

Post reply on HN