Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

facebook.com

Post reply on HN
Print view View original

341–350 of 381 posts

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#341

Post by Dylan16807 »

tripzilch wrote: post #275
SwellJoe wrote: post #231

Earlier quoted context omitted.

So, this is why people don't take you seriously. I know a couple of guys like you in my local activist community, who take a very hostile "I know the truth and you're all fools!" attitude, despite the fact that their audience is mostly very sympathetic to most of their assertions. We know spying on the Internet takes place; HN is full of cantankerous old Internet geeks who've seen, first-hand, plenty of examples of t…

Hey, calling people "schizophrenic" is not cool. Just sayin'.

When describing behavior and not as a slur I don't see any problem.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#342

Post by TomAnthony »

monkeypizza wrote: post #302
rapind wrote: post #296

Earlier quoted context omitted.

Well I do have at least some data to back that up. I know that they are a walled garden that keeps a ton of user data for every account (even the subset represented in Open Graph is significant). And it's with this knowledge that I make my conclusion. * Accounts subsisting of an unusual proportion of ad clicking activity compared to their other activities are likely bots. * New accounts with no history but high ad cl…

Compare it to google's problem filtering out automatically generated splogs, though - it took them years to make progress against that. Real human social activity is a lot more random than real informational content in blogs - so fb's problem seems harder. Also, facebook banning accounts is a much stronger action than google just lowering the pagerank of suspected splogs. So overall I think fb's problem is harder tha…

I come down on the side that it shouldn't be so difficult for Facebook to at least spot some of the 4 bots per human.

However, even if you assume it is a hard problem - thats more reason that they should have listened (and responded!!) when a company identified this problem with some hard data to back it up.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#343

Post by pegmanm »

rapind wrote: post #239
kirubakaran wrote: post #227

Earlier quoted context omitted.

I understand that it is challenging for a third party like you to identify the bots. But Facebook has more than enough signal on each user clicking on an ad (friends, posts, location, 'non ad-click activity' to 'ad-click activity' ratio) to determine if that user is a bot. Looks like they are choosing to not do anything with this information. "FB is turning a blind eye to keep their revenue" seems to stand up to Occa…

Couldn't agree more. It should be almost trivial for them to track down bots and flag bad accounts. I definitely wouldn't classify it as one of their harder problems.

This is a mostly solved problem though. Or at least there are people and products out there that do this specifically for you, not with a focus on bots alone as that is a bit of a edge case but for mobile devices.

DeviceAtlas and WURFL are both products that I have used that attempt to have complete coverage of all UA strings in use in the wild even, the ones that attempt to spoof legitimate UA strings.

While their focus is on device detection and the properties of these devices for mobile content adaptation, knowing which connections are from bots is just as important if you are going to alter what content you serve up based on UA.

These product actively seek out new UAs and add them to their rule sets in a way that I can only assume will be more accurate over time than a home grown solution.

FWIW I know that FB use at least one of these products for their device detection, so if you want to match what FB thinks (knows!?) this is where you would start.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#344

Post by lmm »

dredmorbius wrote: post #291
pbreit wrote: post #268

Earlier quoted context omitted.

Would "white listing" the dozen or so top agents that make up the vast majority of traffic work better?

You'd want to white-list both based on user-agent string and IP / CIDR origin. Keeping those lists maintained would be a bit of fairly constant effort, unless you could come up with a self-training mechanism, or a self-validating mechanism. Killing Google's ability to crawl your site is pretty much as bad or worse than blocking bots. Though you should also be able to ID bad guys by noting which IPs the spoofed user-a…

In the general case it's important to let google crawl your site, but facebook would probably be fine without it.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#345

Post by radicalbyte »

zburt wrote: post #263
ebtalley wrote: post #255

Earlier quoted context omitted.

data would be * IP addresses, one can assume a bot would only have a set of addresses they could use, barring botnets. * request patterns, ie: did the bot request css/js, etc * request timeframes * UA strings Sure, its a big data problem, but I can imagine that Facebook has solved these types of scenarios many times over.

What if you start a new Amazon EC2 spot instance (netting you a new IP address), start up Chromium in headless mode (say, using Xvfb), navigate to the website of choice, use mouse automation to start clicking around, click the ad, spend 5 minutes clicking around in a semi-choreographed pattern on the advertisee's website, and then shut down the instance -- only to repeat? Moreover, Amazon is always buying new IP subn…

Then pay Amazon for a list of their EC2 IPs, or obtain that information from a public source (i.e. RIPE, ARIN).

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#346

Post by fsniper »

freeall wrote: post #322
don_draper wrote: post #290

Earlier quoted context omitted.

One quick way to detect bots is to look for a GET request of the robots.txt file

Sort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to look through robots.txt.

That's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#347

Post by lifeisstillgood »

kirubakaran wrote: post #227
nhashem wrote: post #162

My startup is essentially an advertising aggregator (pooling traffic from a variety of publishers and routing it to advertisers) and dealing with things like bot detection is a HUGE chunk of what we work on, technology-wise. Let me try and give you an idea of how deep the rabbit hole can go. - Okay, you want to detect bots. Well, "good" bots usually have a user agent string like, "Mozilla/5.0 (compatible; Googlebot/2…

I understand that it is challenging for a third party like you to identify the bots. But Facebook has more than enough signal on each user clicking on an ad (friends, posts, location, 'non ad-click activity' to 'ad-click activity' ratio) to determine if that user is a bot. Looks like they are choosing to not do anything with this information. "FB is turning a blind eye to keep their revenue" seems to stand up to Occa…

Just as a thought - facebook must be in a unique position to provide almost unbeatable captchas - just put up images of a known friend and four / forty random pictures - the one who chooses correct is likely the person logged in

ignore this is fb already does it

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#348

Post by qntm »

nhashem wrote: post #162

My startup is essentially an advertising aggregator (pooling traffic from a variety of publishers and routing it to advertisers) and dealing with things like bot detection is a HUGE chunk of what we work on, technology-wise. Let me try and give you an idea of how deep the rabbit hole can go. - Okay, you want to detect bots. Well, "good" bots usually have a user agent string like, "Mozilla/5.0 (compatible; Googlebot/2…

> How about we just make a "blacklist" of these known bots, look up every user agent, and compare against the blacklist? So now every single request to your site has to do a substring match against every single term in this list. Depending on your site's implementation, this is probably not trivial to do without taking some sort of performance hit.

Build a finite state machine which only accepts the terms in the blacklist. That should be a one-time operation.

Then feed each request into the FSM and see if you get a match. Execution time is linear in the length of the request, regardless of the number of terms in the blacklist.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#349

Post by lifeisstillgood »

kirubakaran wrote: post #227
nhashem wrote: post #162

My startup is essentially an advertising aggregator (pooling traffic from a variety of publishers and routing it to advertisers) and dealing with things like bot detection is a HUGE chunk of what we work on, technology-wise. Let me try and give you an idea of how deep the rabbit hole can go. - Okay, you want to detect bots. Well, "good" bots usually have a user agent string like, "Mozilla/5.0 (compatible; Googlebot/2…

I understand that it is challenging for a third party like you to identify the bots. But Facebook has more than enough signal on each user clicking on an ad (friends, posts, location, 'non ad-click activity' to 'ad-click activity' ratio) to determine if that user is a bot. Looks like they are choosing to not do anything with this information. "FB is turning a blind eye to keep their revenue" seems to stand up to Occa…

Just as a thought - facebook must be in a unique position to provide almost unbeatable captchas - just put up images of a known friend and four / forty random pictures - the one who chooses correct is likely the person logged in

ignore this is fb already does it

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#350

Post by ars »

lifeisstillgood wrote: post #347
kirubakaran wrote: post #227

Earlier quoted context omitted.

I understand that it is challenging for a third party like you to identify the bots. But Facebook has more than enough signal on each user clicking on an ad (friends, posts, location, 'non ad-click activity' to 'ad-click activity' ratio) to determine if that user is a bot. Looks like they are choosing to not do anything with this information. "FB is turning a blind eye to keep their revenue" seems to stand up to Occa…

Just as a thought - facebook must be in a unique position to provide almost unbeatable captchas - just put up images of a known friend and four / forty random pictures - the one who chooses correct is likely the person logged in ignore this is fb already does it

They do in fact do that (or something like it) when you need to recover an account/password.
Post reply on HN