Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

191–200 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#191

Post by fennecfoxen »

mulmen wrote: post #156
ortusdux wrote: post #94

Earlier quoted context omitted.

My office still gets spam faxes to this day. The paper and toner only add up to a few cents a month, so it's not worth doing anything about. I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a sta…

My desk phone at an old job used to get dialed by a fax machine. Not fun picking that up. I redirected it to a virtual fax line and it turns out it was a local clinic faxing medical records. I faxed them back with some message about you have the wrong number but they never stopped.

If you really wanted to make them stop, you might find a contact for their lawyer and make HIPAA noises at them ;)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#192

Post by TheSpiceIsLife »

oh_sigh wrote: post #187
TheSpiceIsLife wrote: post #180

Earlier quoted context omitted.

Boo hiss. There’s plenty of room here for a polite exchange between professionals. Collapse the thread and move on.

The question is why does everyone need to read what really could first be something done on a private communication.

No body is forcing you to read anything.

Though I will concede: reading is involuntary to literate adults.

Having said that, there’s currently, what? 28 comments in this thread.

Is that indicative enough that at least some people have derived value from it?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#193

Post by jimmaswell »

Drakim wrote: post #185
jimmaswell wrote: post #177

Earlier quoted context omitted.

Worse than unimportant, I find copyleft harmful to the developer community as a whole. Software is hardly "free" when you have to bend to the author's demands to license your entire rest of your project under their ideology if you want to use it, and I think we should stop calling it so. Maybe "Conditionally Free Software" instead. You're hardly helping the world releasing libraries in terms that nobody but hobbyists…

Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.

Users don't care about any of that, but developers are certainly hindered by restrictive licenses, which in turn hurts users.

I never found a right to see source code compelling as a real right. To read the assembly and modify something they bought, sure, but not an entitlement to the source.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#194

Post by justinclift »

_lqaf wrote: post #33
leafo wrote: post #9

Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834 The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off. Of course, sample size of 1, etc. It could have been coincidental.

Interesting. Implemented DROP for some Cloudflare net blocks a few weeks ago due to lots of weird interrupted TCP connections from them.

That's seemed to be good enough, but will consider the TARPIT option if it turns out to be needed. :)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#195

Post by maxk42 »

ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

GPL license states you must provide source code to the end-user. In this case, the end-user is legally Napolux, not the visitors to his site.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#196

Post by cjslep »

Drakim wrote: post #185
jimmaswell wrote: post #177

Earlier quoted context omitted.

Worse than unimportant, I find copyleft harmful to the developer community as a whole. Software is hardly "free" when you have to bend to the author's demands to license your entire rest of your project under their ideology if you want to use it, and I think we should stop calling it so. Maybe "Conditionally Free Software" instead. You're hardly helping the world releasing libraries in terms that nobody but hobbyists…

Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.

It may focus on user's rights, but it still requires technical expertise to exercise half of the 4 freedoms: #1 (inspection & modification) and #3 (distributing your modifications). Merely knowing to ask "can I see the source code" I would put into the "technical user" realm. Overwhelmingly, most people don't know or don't care.

The other two freedoms #0 (freedom to run) and #2 (freedom to share) are readily obvious to non-technical users. "Double click to run" and "drag and drop to copy". Unfortunately, they are also often permitted by non-free/libre software. So non-technical users that can readily see they can do #0 and #2 and generally have no litmus test to further determine whether the software is "free/libre".

This is a case where the ideology's practical concerns hamper its purity. I critique despite generally liking the FLOSS ideal, but it's important to know its flaws.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#197

Post by rocho »

apocalyptic0n3 wrote: post #126
hinkley wrote: post #42

Earlier quoted context omitted.

Tarpits are an underappreciated solution to a pool of bad actors. You can add artificial wait times to responses, or you can just route all of the 'bad' traffic to one machine, which becomes oversubscribed (be sure to segregate your stats!). All bad actors fighting over the same scraps creates proportional backpressure. Just adding 2 second delays to each request won't necessarily achieve that if multiple user agents…

I never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval t…

Be careful, as tarpitting connections can consume your resources faster than those of the attacker.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#198

Post by golem14 »

napolux wrote: post #132
tombrossman wrote: post #124

Earlier quoted context omitted.

My "404-pests" fail2ban-client status, which drops everything making *.php requests (This machine has never had PHP installed...): 2a03:2880:10ff:14::face:b00c 2a03:2880:10ff:21::face:b00c 2a03:2880:11ff:1a::face:b00c 2a03:2880:11ff:1f::face:b00c 2a03:2880:11ff:2::face:b00c 2a03:2880:12ff:10::face:b00c 2a03:2880:12ff:1::face:b00c 2a03:2880:12ff:9::face:b00c 2a03:2880:12ff:d::face:b00c 2a03:2880:13ff:3::face:b00c 2a03…

I think we're together in this, my friend

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#199

Post by ThrustVectoring »

PetahNZ wrote: post #161
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

My understanding is unless he's distributing it (not just serving it from a website) he doesnt need to release his changes. That's what the AGPL is for.

It's not the copyright on the server code that is at issue, but the copyright on the HTML and CSS files (and portions thereof) that get distributed by the server.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#200

Post by jschwartzi »

justinclift wrote: post #194
_lqaf wrote: post #33

Earlier quoted context omitted.

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off. Of course, sample size of 1, etc. It could have been coincidental.

Interesting. Implemented DROP for some Cloudflare net blocks a few weeks ago due to lots of weird interrupted TCP connections from them. That's seemed to be good enough, but will consider the TARPIT option if it turns out to be needed. :)

one man’s weird interrupted TCP connection is another man’s nmap attempt.
Post reply on HN