Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

241–250 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#241

Post by detaro »

rapind wrote: post #231
paulpauper wrote: post #190

Earlier quoted context omitted.

twitter does the same thing. it sends a bunch of spoofed US visitors from Korea and Germany and US. The bots are spoofed to make it harder to filter them.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later).

Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., although you normally can fit an identifier of what you really are at the end. (As an example, here's an iOS Safari user agent: "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1" - it's not Mozilla, it's not Gecko, but it has those keywords and patterns because sites expect to see that)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#242

Post by JadeNB »

systemvoltage wrote: post #35
crazygringo wrote: post #20

Earlier quoted context omitted.

I disagree completely, I find it extremely useful. Most links are unreadable (e.g. an ID to a cloud file), and are truncated anyways even if readable. The preview gives me the title of the webpage, which is infinitely more useful -- especially letting me know whether it's a document I've already seen (and don't need to open) or something new, and if so, what.

To me it pollutes the discussion away from the messages and puts a bunch of thumbnails where it could just be a clean text conversation. The thing that bothers me is also that I, as a user, have no choice - it just does it automatically. Do you like IRC?

I suspect that most apps have a way to turn this off. Slack certainly does. That doesn't stop other users from seeing previews of your links, if it's set up that way for them, but it does prevent you from seeing previews of others' (and your own) links.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#243

Post by JadeNB »

katbyte wrote: post #62
systemvoltage wrote: post #35

Earlier quoted context omitted.

To me it pollutes the discussion away from the messages and puts a bunch of thumbnails where it could just be a clean text conversation. The thing that bothers me is also that I, as a user, have no choice - it just does it automatically. Do you like IRC?

this is one of those "people like different things" situations and a config option should be added to support both.

Slack, and I suspect most other messaging apps, has such a config option.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#244

Post by jimmaswell »

dahfizz wrote: post #206
jimmaswell wrote: post #193

Earlier quoted context omitted.

Users don't care about any of that, but developers are certainly hindered by restrictive licenses, which in turn hurts users. I never found a right to see source code compelling as a real right. To read the assembly and modify something they bought, sure, but not an entitlement to the source.

Nobody is forcing you to use GPL code. It is stinking of entitlement to demand that you get to use other people's code regardless of what they think about it.

I don't feel entitled to it, I just consider the prevalence of copyleft bad for the developer community. License however you want, doesn't mean I'm entitled for criticizing it.

What might be stinking of entitlement is the idea users have an inalienable right to source code.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#245

Post by Aeolun »

jasonlotito wrote: post #183
thelean12 wrote: post #172

Earlier quoted context omitted.

Come on, this is super petty. And that includes your license choice and your enforcement for this.

So, just to be clear, you are perfectly fine with someone taking something someone else created and violating the terms upon which they were given that thing? e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that?

> e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that?

If I released it on Github, under any license whatever? I’d more or less be expecting that.

If it was about the 4hr of work that went into my blog theme, I wouldn’t be bothered at all.

But then, I wouldn’t release anything like that under the GPL.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#246

Post by TheChaplain »

Reading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes.

Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#247

Post by jimmaswell »

Dahoon wrote: post #201
jimmaswell wrote: post #177

Earlier quoted context omitted.

Worse than unimportant, I find copyleft harmful to the developer community as a whole. Software is hardly "free" when you have to bend to the author's demands to license your entire rest of your project under their ideology if you want to use it, and I think we should stop calling it so. Maybe "Conditionally Free Software" instead. You're hardly helping the world releasing libraries in terms that nobody but hobbyists…

You sound very bitter that a developers that let others use their code gets to pick his license of choice. Should everything by locked down like Microsoft Windows code or an Apple phone? If what you want is for everything to be completely free from licenses instead, just code your own version and release it as freeware. If you only complain and don't then you are just being hypocritical. >You're hardly helping the wo…

I'm not bitter, just disappointed in all the wasted developer time that happens because people get caught up in these copyleft ideas. Of course you can pick whatever license you want but that doesn't mean it can't be criticized.

> If you only complain and don't then you are just being hypocritical.

You don't need to be an architect to complain about crumbling bridges, but indeed I have released software under more free licenses.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#248

Post by tjalfi »

mulmen wrote: post #156
ortusdux wrote: post #94

Earlier quoted context omitted.

My office still gets spam faxes to this day. The paper and toner only add up to a few cents a month, so it's not worth doing anything about. I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a sta…

My desk phone at an old job used to get dialed by a fax machine. Not fun picking that up. I redirected it to a virtual fax line and it turns out it was a local clinic faxing medical records. I faxed them back with some message about you have the wrong number but they never stopped.

My dad used to receive a lot of misdirected faxes.

His solution was to send a return fax with disorderly handwriting begging and pleading for them to fax someone else.

The other person stopped faxing him.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#249

Post by qroshan »

cdown wrote: post #89
napolux wrote: post #65

Earlier quoted context omitted.

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

yeah, for all you know, it could just a naive/rogue employee building a crawler for his personal/20% project

Re: A Facebook crawler was making 7M requests per day to my stupid website

#250

Post by gerdesj »

pera wrote: post #168
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

Nice idea, I like the thinking. I'll tuck that away for use later. PowerDNS has LUA built in amongst a few other things.

My stack of projects to do is growing at a hell of a rate and I'm not popping them off the stack fast enough.

Post reply on HN