Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebooks crawls every page recorded by its tracking pixel

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Facebooks crawls every page recorded by its tracking pixel

#61

Post by throwaway2016a »

dspillett wrote: post #45
smt88 wrote: post #32

Earlier quoted context omitted.

This sounds a little extreme at first, but I actually totally agree. It's in murky waters when it comes to GDPR, for starters. Where do they draw the line? Why not run a keylogger through embedded like buttons and widgets? That sounds worse, but isn't all that much worse.

> It's in murky waters when it comes to GDPR, for starters. I'm not sure about from facebook's side, but from the point of view of how GDPR applies to the side being crawled if they, as custodians of PII and other sensitive data, are handing it out to unauthenticated requests, they might be liable for punishment for lack of due diligence.

I agree with this. The website author is potentially liable for providing inadequate protections to the user's PII. I don't see anything that would implicate Facebook here.

Although, there is an interesting side effect that applies to all crawlers in that website owners failing to protect their customer PII like this means that crawlers inadvertently gather and store personal data as a side effect. I can't help but wonder if there is some liability there and if there is if something like AI or pattern matching can help to scrub the info before it is stored.

Re: Facebooks crawls every page recorded by its tracking pixel

#62

Post by dspillett »

throwaway2016a wrote: post #61
dspillett wrote: post #45

Earlier quoted context omitted.

> It's in murky waters when it comes to GDPR, for starters. I'm not sure about from facebook's side, but from the point of view of how GDPR applies to the side being crawled if they, as custodians of PII and other sensitive data, are handing it out to unauthenticated requests, they might be liable for punishment for lack of due diligence.

I agree with this. The website author is potentially liable for providing inadequate protections to the user's PII. I don't see anything that would implicate Facebook here. Although, there is an interesting side effect that applies to all crawlers in that website owners failing to protect their customer PII like this means that crawlers inadvertently gather and store personal data as a side effect. I can't help but w…

facebook might have an issue with having collected the data too, of course, but the source site definitely should be taking appropriate measures to avoid handing it out in the first place.

Re: Facebooks crawls every page recorded by its tracking pixel

#63

Post by greenone »

dna_polymerase wrote: post #35

It is the fucking internet, if you put something on there you should expect someone to find it, be it a crawler or an attacker. > 1. they are crawling potentially sensitive information granted by links with tokens If tokens in GET params are your security concept: please leave the entire field. 2. they are triggering potentially harmful and/or confusing actions in your website by repeating links So you built somethin…

to clarify:

- marketing wants some tracking, some developers adds it

- ecommerce websites in the real world tend to "need" these tracking/conversion codes

- you do have legitimate get-requests like password-reset links with tokens, also we do use payment providers who send the customers back to us with get links which include payment tokens, newsletter-unsubscribe links are also often simple token links

- and yes normally a get-request should not change anything (at least not when its just repeated) but the sheer fact that they have access to it _and_ are crawling it is bad

my point being that I find it that they would just crawl everything they recorded instead of just crawling pages which are linked publicly or which are targeted in ad-campaigns combined with the fact that they don't warn you about it

Re: Facebooks crawls every page recorded by its tracking pixel

#64

Post by greenone »

eli wrote: post #56
CapacitorSet wrote: post #55

Earlier quoted context omitted.

>websites they were essentially invited to Using an analytics pixel is _not_ an invitation to crawl a website.

No, merely posting it on a public server was the invitation

thats like saying "having a public website is an invitation to DOS-attacks"

there are conventions and reasonable expectations, until now I did not expect that a tracking-pixel would be the basis for crawling, so far most crawlers tend to crawl whats publicly linked, not whats potentially publicly reachable if one knows every url there is

Re: Facebooks crawls every page recorded by its tracking pixel

#65

Post by laumars »

greenone wrote: post #63
dna_polymerase wrote: post #35

It is the fucking internet, if you put something on there you should expect someone to find it, be it a crawler or an attacker. > 1. they are crawling potentially sensitive information granted by links with tokens If tokens in GET params are your security concept: please leave the entire field. 2. they are triggering potentially harmful and/or confusing actions in your website by repeating links So you built somethin…

to clarify: - marketing wants some tracking, some developers adds it - ecommerce websites in the real world tend to "need" these tracking/conversion codes - you do have legitimate get-requests like password-reset links with tokens, also we do use payment providers who send the customers back to us with get links which include payment tokens, newsletter-unsubscribe links are also often simple token links - and yes nor…

> my point being that I find it that they would just crawl everything they recorded instead of just crawling pages which are linked publicly or which are targeted in ad-campaigns

There's no way to know which pages are linked publicly without crawling every page for links. So you're right back at square one.

Ultimately if it's on a Internet-facing web server and not hidden behind an IP whitelist or secure login function then you have to assume it is public. All you are arguing is about different degrees of "public" which somewhat misses the real issue of website security.

Some crawlers do deliberately hit random URLs to check how you're handling 404s. Over crawlers are entirely dishonest and will try to find content that wasn't intended to be made public. How are you going to handle them if you're stumped with the Facebook crawlers that you invited onto your site?

> ...combined with the fact that they don't warn you about it

It's pretty obvious behavior in my opinion but maybe they could have been more explicit. However going back to my previous point, no other crawler advertises what it's going to crawl beforehand. So where do you draw the line? Ranting that Google indexed your site? What about visitors buying stuff on your ecommerce package without prior communication requesting access to the site?

You wouldn't ask customers in a bricks-and-mortar store to state their intentions the moment they walked through the shop door so why should every HTTP user agent have to do the same? While web security can be both complex and maddening, responsibility of hardening the site is still yours; not Facebook's.

Re: Facebooks crawls every page recorded by its tracking pixel

#68

Post by eli »

greenone wrote: post #64
eli wrote: post #56

Earlier quoted context omitted.

No, merely posting it on a public server was the invitation

thats like saying "having a public website is an invitation to DOS-attacks" there are conventions and reasonable expectations, until now I did not expect that a tracking-pixel would be the basis for crawling, so far most crawlers tend to crawl whats publicly linked, not whats potentially publicly reachable if one knows every url there is

Posting a file to a public web server is an implicit invitation for clients (human or automated) to download that file. That's why "secret urls" are universally considered to provide very little security.

There are common conventions (not always followed) around robots.txt and what files to crawl, but I'm not aware of any rules or conventions or standards around URL discovery. Plenty of crawlers attempt to crawl every registered domain name, for example.

"DOS Attack" is sort of a loaded term since it implies malice. Clearly running a web server doesn't mean you invite malicious attacks (though perhaps you should expect them). Some people consider Googlebot to be a DOS attack since it can easily bring poorly designed sites to their knees.

Re: Facebooks crawls every page recorded by its tracking pixel

#69

Post by eli »

thecatspaw wrote: post #59
eli wrote: post #56

Earlier quoted context omitted.

No, merely posting it on a public server was the invitation

It is not. Leaving your door unlocked is not inviting everybody in to take your stuff. You might make it easier for them to break in, but it still is a break in. Making something available to the public is not the same as going to the google webmaster tools and telling them to index your page.

Someone viewing a webpage you put online is not at all like someone stealing something you own.

Re: Facebooks crawls every page recorded by its tracking pixel

#70

Post by freehunter »

thecatspaw wrote: post #58
K0nserv wrote: post #28

Earlier quoted context omitted.

I believe the tracking stuff comes with Like buttons and other Facebook widgets those are what I am referring to when I say "add value". But it could be argued that the tracking alone never adds value to the user.

It could also be argued that not even the like buttons add value

It could also be argued the exact opposite.
Post reply on HN