Live data from Hacker News

Facebooks crawls every page recorded by its tracking pixel

news.ycombinator.com

31–40 of 75 posts

Re: Facebooks crawls every page recorded by its tracking pixel

#31
post #17
post #12

I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addresses (millions of hits daily, 50+ for the same URL at times). They don't even set the User-Agent header. There's a bug report regarding the missing header here: https://developers.facebook.com/bugs/1654459311255613/ Unfortunately it seems impossible to get in t…

Send a cease and desist to the CTO. Wait 30 days, then sue under the CFAA. LinkedIn did it

No, LinkedIn was sued by the recipient of the letter.

Re: Facebooks crawls every page recorded by its tracking pixel

#32
post #9

This is borderline criminal. Practically CSRF attack.

This sounds a little extreme at first, but I actually totally agree. It's in murky waters when it comes to GDPR, for starters.

Where do they draw the line? Why not run a keylogger through embedded like buttons and widgets? That sounds worse, but isn't all that much worse.

Re: Facebooks crawls every page recorded by its tracking pixel

#33
> we figured out that facebooks Facebot crawler will crawl _every_ url that was recorded by their tracking pixel.

I would be more surprised to find out that they didn't crawl everything they can, specifically pages that invite them in.

> 1. they are crawling potentially sensitive information granted by links with tokens

If the page contains sensitive information you absolutely should not have code that you do not control (any code loaded from third party hosts, not just facebook's bits).

As a matter of security due diligence if you have third party hosted code linked into any such pages you should remove it with some urgency and carefully review the design decisions that lead to the situation. If you really must have the third party code in that area then you'll need to find a way of removing the need for the tokens being present.

Furthermore, if the information is sensitive to a particular user then your session management should not permit a request from facebook (or any other entity that has not correctly followed your authentication procedure) to see the content anyway.

> 2. they are triggering potentially harmful and/or confusing actions in your website by repeating links

Possibly true, but again that suggests a design flaw in the page in question. I assume that they are not sending POST or PUT requests? GET and HEAD requests should at very least be idempotent (so repeated calls are not a problem) and ideally lack any lasting side effect (with the exception of logging).

> 3. they are repeating requests in a broken way by not encoding url-parameters correctly

That does sound like a flaw, but one that your code should be immune to being broken by. Inputs should always be verified and action not taken unless they are valid. This is standard practise for good security and stability. The Internet is a public place, the public includes both deliberately nasty people and damagingly stupid ones so your code needs to take proper measures to not allow malformed inputs to cause problems.

You can't use "the page isn't normally linked from other sources so won't normally be found by a crawler" as a valid mitigation because the page could potentially be found by a malicious entity via URL fuzzing.

> 4. I could not find a warning or note on their tracking-pixel documentation that pages tracked would be crawled later

A warning would be nice, but again unless they explicitly say they won't do such things I would be surprised to find that they didn't not that they do.

Re: Facebooks crawls every page recorded by its tracking pixel

#34
post #3

Shocking! Abuse of power and shady tracking techniques by Facebook? Unheard of! Seriously, this cannot be surprising after learning that the Messenger app listens to everything you do, all the time. That's just off the top of my head. They are doing this and much more.

On iOS this is simply not possible without the user being explicitly notified.

Can you provide some evidence of this happening on Android ?

Also Facebook categorically denies this: http://www.bbc.com/news/technology-41776215

Re: Facebooks crawls every page recorded by its tracking pixel

#35
It is the fucking internet, if you put something on there you should expect someone to find it, be it a crawler or an attacker.

> 1. they are crawling potentially sensitive information granted by links with tokens

If tokens in GET params are your security concept: please leave the entire field.

2. they are triggering potentially harmful and/or confusing actions in your website by repeating links

So you built something that can be triggered by a simple HTTP request and may have a harmful potential? Wow.

3. they are repeating requests in a broken way by not encoding url-parameters correctly

You are kidding right? That's a problem to you? Either your Webserver drops these or your routes don't match, end of story.

4. I could not find a warning or note on their tracking-pixel documentation that pages tracked would be crawled later

Not a problem, you put it on the web and it will be crawled. Did you ever use Chrome? They report every URL you type to the Google Crawler. Read that anywhere lately?

Re: Facebooks crawls every page recorded by its tracking pixel

#37
post #18
post #5

> 1. they are crawling potentially sensitive information granted by links with tokens Don't put Facebook tracking on sensitive pages. Actually as a service to your users don't put it anywhere where it doesn't add value. > 2. they are triggering potentially harmful and/or confusing actions in your website by repeating links They only perform idempotent[0]* requests which should not have any negative effect if performe…

GET is only idempotent in theory. Way too many people abuse GET when creating websites.

Those people should suffer the consequences.

I'm not a fan of facebook in the slightest, but they are crawling websites they were essentially invited to.

Re: Facebooks crawls every page recorded by its tracking pixel

#38
A while ago while looking at the apache logs I noticed that the AdWords remarketing pixel does the same, it was trying to crawl private URLs that are only accessible to 'admins' that are not linked publicly. I'm not sure if this is still valid as I blocked by using robots.txt.

Also, the same crawler ignores the "User-agent: *" directive in the robots.txt file and you have to add specific rules for it: "User-agent: Adsbot-Google"

Re: Facebooks crawls every page recorded by its tracking pixel

#39
post #9

This is borderline criminal. Practically CSRF attack.

You might be able to argue that, though you are arguing against accepted practise (are you are wanting to ban all web crawling?).

While two wrongs don't make a right, assuming we accept that facebook is wrong in this instance which I don;t think I do, the code for the page handing out sensitive information to an unauthenticated request or taking action based on malformed inputs is negligent.

"Information wants to be free" is not just a hippie ideal it is a technical warning. Unless you take proper measures to control and protect sensitive data it will find a way out.

Post reply on HN