Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

231–240 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#231

Post by rapind »

paulpauper wrote: post #190
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

twitter does the same thing. it sends a bunch of spoofed US visitors from Korea and Germany and US. The bots are spoofed to make it harder to filter them.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#232

Post by noir-york »

rocho wrote: post #197
apocalyptic0n3 wrote: post #126

Earlier quoted context omitted.

I never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval t…

Be careful, as tarpitting connections can consume your resources faster than those of the attacker.

Only if you use conntracking though no?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#233

Post by TheSpiceIsLife »

napolux wrote: post #222
TheSpiceIsLife wrote: post #180

Earlier quoted context omitted.

Boo hiss. There’s plenty of room here for a polite exchange between professionals. Collapse the thread and move on.

Come on. I love Ruud's posts and I took inspiration for my blog. He was right asking for a link, which I gladly provided. That's it for me.

Sorry, I think you might have misinterpreted my comment?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#234

Post by lsiebert »

errantmind wrote: post #214
ThrustVectoring wrote: post #199

Earlier quoted context omitted.

It's not the copyright on the server code that is at issue, but the copyright on the HTML and CSS files (and portions thereof) that get distributed by the server.

It is my understanding that style-related code is effectively not copyrightable in the US. Is my knowledge out of date?

NAL, but the resulting style isn't, as there are many ways to generate the same style, but the specific code is.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#235

Post by dang »

apolymath wrote: post #230
ruuda wrote: post #167

Earlier quoted context omitted.

Thanks, I’m flattered to see it be used as inspiration :) I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

Flattered that your few, boring lines of CSS and HTML (that could easily be reproduced by a monkey) was used on someone else's website?

We've banned this account for breaking the site guidelines.

If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future.

https://news.ycombinator.com/newsguidelines.html

Re: A Facebook crawler was making 7M requests per day to my stupid website

#236

Post by DJBunnies »

apolymath wrote: post #230
ruuda wrote: post #167

Earlier quoted context omitted.

Thanks, I’m flattered to see it be used as inspiration :) I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

Flattered that your few, boring lines of CSS and HTML (that could easily be reproduced by a monkey) was used on someone else's website?

That isn't how you hacker news.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#237

Post by ruuda »

napolux wrote: post #220
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

Hello there. I've added a link to your website in the footer! :) https://coding.napolux.com/ Thanks for reaching out!

Awesome, thank you! Sorry for derailing your thread, I did not mean for my comment to escalate like it did.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#238

Post by WealthVsSurvive »

yupyup54133 wrote: post #115
gknoy wrote: post #109

Earlier quoted context omitted.

Really? I felt that what happened to Aaron was a tragic injustice on many levels (from the fact that he was charged at all to the number of charges they threw at him), but for some reason I had always thought that it was fairly well-known that that was his intent. I have no idea where I got that impression from, but I do recall reading several articles about him, as well as his blog around that time. It's possible I…

IIRC, he had published a manifesto about how he believed that information that was paid for with public funds should be free for the public to view. However, I don't believe that he ever mentioned why he was making get requests to JSTOR's servers and I know that he never uploaded any of those JSTOR documents to the internet for public download.

Yeah, but any decent criminal lawyer could tear "he once wrote a manifesto" to shreds. His political beliefs would only be pertinent if he wrote the manifesto somehow in connection with what he was doing; otherwise, it would not be sufficient to show intent to distribute.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#239

Post by naikrovek »

ruuda wrote: post #184
RhodesianHunter wrote: post #142

Earlier quoted context omitted.

Despite how it sounds, I ask this with zero judgment and pure curiosity. Why do you care?

I made my site open source so others who like it can take a look at how it works, or give it their own twist. Visitors of my site can see that they have that freedom, but visitors of an adaptation might not know if it’s not stated anywhere.

You're not wrong to request attribution. I don't know why you're getting so much flak.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#240

Post by Nextgrid »

xmprt wrote: post #179
Nextgrid wrote: post #174

Earlier quoted context omitted.

It’s because we as an industry decided to give in and submit to the cloud providers’ bullshit model of paying overpriced amounts for bandwidth while good old bare-metal providers still offer unmetered bandwidth for very reasonable prices.

The question isn't about pricing. The fact that you pay for your own bandwidth and metal is a problem.

Paying for your bandwidth and metal isn't a problem per-se as long as prices are reasonable (which they are with the old-school bare-metal providers). After all, those services do cost money to provide.

The problem happens when prices are extortionate (or the pricing model is predatory, ie pay per MB transferred instead of a flat rate per 1Gbps link) and relatively minor traffic translates to a major bill.

This particular issue is about 80 requests/second which is a drop in the bucket on a 1Gbps link. It would be basically unnoticeable if it wasn't for the cloud providers nickel & diming their customers.

Post reply on HN