Facebooks crawls every page recorded by its tracking pixel • Hacker News Board

Re: Facebooks crawls every page recorded by its tracking pixel

#71

Post by freehunter » Fri, Feb 09, 2018, 4:03 PM UTC

thecatspaw wrote: post #59

eli wrote: post #56

Earlier quoted context omitted.

No, merely posting it on a public server was the invitation

It is not. Leaving your door unlocked is not inviting everybody in to take your stuff. You might make it easier for them to break in, but it still is a break in. Making something available to the public is not the same as going to the google webmaster tools and telling them to index your page.

You can argue it's abuse or illegal or fraud or whatever you want, but here's the thing: how are you going to stop them? Sure, maybe you stop Facebook with a lawsuit... but everyone else is still doing it, even people outside of your legal jurisdiction. They're still going to do it, so it's up to you to stop them with your design. If someone breaks into your website and destroys a user's data or steals their credit card, that user is not going to want to hear "but what they did was against our ToS!"

This isn't your house where there are police patrolling and ready to respond at a moment's notice when they're called. This is the Internet, accessible by almost literally everyone on the planet, and they don't give a shit about your policy. That's why best practices and application security was invented. So use it.

Re: Facebooks crawls every page recorded by its tracking pixel

#72

Post by freehunter » Fri, Feb 09, 2018, 4:04 PM UTC

greenone wrote: post #64

eli wrote: post #56

Earlier quoted context omitted.

No, merely posting it on a public server was the invitation

thats like saying "having a public website is an invitation to DOS-attacks" there are conventions and reasonable expectations, until now I did not expect that a tracking-pixel would be the basis for crawling, so far most crawlers tend to crawl whats publicly linked, not whats potentially publicly reachable if one knows every url there is

I watch my site's Google index and I can tell you 100% I never gave Google explicit permission to crawl 90% of the pages that show up there.

Re: Facebooks crawls every page recorded by its tracking pixel

#73

Post by radicalbyte » Fri, Feb 09, 2018, 6:45 PM UTC

tpllaha wrote: post #16

radicalbyte wrote: post #15

Earlier quoted context omitted.

That's because our industry is full of hopelessly under qualified people who somehow manage to create software by randomly throwing together bits of code from stackoverflow / books and tweaking it until it "works".

That's true. But also trackers (especially big ones) deliberately try to make it as easy as possible to accidentally include them in your page. e.g: the if you use their cdn for fonts/style sheets, if you include a fb like button etc. A friend of mine covers this more extensively in this blog post which I found a very interesting read: https://remusao.github.io/posts/static-comments.html So I think it's still concern…

Yeah, of course they'll try that. They're businesses, they make their money by tracking people. When I inherit a team of project these CDN links are the second thing to get removed / fixed (after their the inevitable unencrypted passwords / homerolled security and homerolled SQL).

Re: Facebooks crawls every page recorded by its tracking pixel

#74

Post by Quppa » Sat, Feb 10, 2018, 1:22 AM UTC

AznHisoka wrote: post #50

Quppa wrote: post #12

I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addresses (millions of hits daily, 50+ for the same URL at times). They don't even set the User-Agent header. There's a bug report regarding the missing header here: https://developers.facebook.com/bugs/1654459311255613/ Unfortunately it seems impossible to get in t…

what site do you own (if you can tell)?

Not sure I can say.

On a positive note, it's given us an opportunity to focus on performance improvements :)

Re: Facebooks crawls every page recorded by its tracking pixel

#75

Post by Piskvorrr » Thu, Feb 15, 2018, 9:19 AM UTC

thecatspaw wrote: post #59

eli wrote: post #56

Earlier quoted context omitted.

No, merely posting it on a public server was the invitation

It is not. Leaving your door unlocked is not inviting everybody in to take your stuff. You might make it easier for them to break in, but it still is a break in. Making something available to the public is not the same as going to the google webmaster tools and telling them to index your page.

"Hello, I am a HTTP client, can I have /some/super/secret/page?" "200 OK, here it is"

That's your server complying with the request. Whether by intent or by oversight, doesn't matter: the client comes and asks, and your server can refuse. If it complies, well, you told it to. Whether you have merely exposed the page to the public or also shouted its URL from the rooftops, that's completely irrelevant. If it's not supposed to be public, don't make it public.

"Hello, I am a HTTP client, can I have /some/super/secret/page?" "Oh, but you are ^User-agent$=.acebook ? Nope, 403 Forbidden, no data for you." (Or, more generally, "And who are you? 401 Authorize!" - or any other sort of mandatory access control)