Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

facebook.com

Post reply on HN
Print view View original

291–300 of 381 posts

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#291

Post by dredmorbius »

pbreit wrote: post #268
nhashem wrote: post #162

My startup is essentially an advertising aggregator (pooling traffic from a variety of publishers and routing it to advertisers) and dealing with things like bot detection is a HUGE chunk of what we work on, technology-wise. Let me try and give you an idea of how deep the rabbit hole can go. - Okay, you want to detect bots. Well, "good" bots usually have a user agent string like, "Mozilla/5.0 (compatible; Googlebot/2…

Would "white listing" the dozen or so top agents that make up the vast majority of traffic work better?

You'd want to white-list both based on user-agent string and IP / CIDR origin.

Keeping those lists maintained would be a bit of fairly constant effort, unless you could come up with a self-training mechanism, or a self-validating mechanism.

Killing Google's ability to crawl your site is pretty much as bad or worse than blocking bots. Though you should also be able to ID bad guys by noting which IPs the spoofed user-agents are coming from to, say, some honey-pot links specifically disallowed for that user-agent in your robots.txt file.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#292

Post by belgianguy »

Relevant experiment done by the BBC: http://www.bbc.com/news/technology-18819338

(paraphrased) "Virtual Bagels": A company that does nothing gathers 4000 likes in 4 days. When set to Worldwide, it soared from hits in Egypt, Indonesia and the Philippines. When limited to the UK the needle barely moved.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#293

Post by GoodIntentions »

flexie wrote: post #126
Cushman wrote: post #80

Earlier quoted context omitted.

> (or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%). That's actually an interesting proposal-- 100% of the time when I click on an ad, I didn't mean to and I try to close it before it has a chance to annoy me. Probably doesn't make sense on a large scale, but it's something to consider.

If that is true and 70-80 percent hit the back button before the js files load, it still says a lot about the quality of the clicks from the perspective of someone considering advertising on FB...

so the net result would be the same.

Someone's grandmother miss-clicking and then mashing back is about as useful to the advertiser as a bot. conversions are where it's at. Poor quality traffic / bots / it's all the same. the advertiser quits paying

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#294

Post by mmaunder »

The data here is clear. 80% of clicks are from non JS clients. There's no getting around this. I run an ad network, a real-time analytics product and a WordPress security product that provides real-time traffic and we use JS vs non JS to distinguish between human and non-human for our customers and it is 95+% reliable.

As an ad network provider it's your responsibility to filter spam clients based on heuristics like click frequency from a geo location, JS support in clients, behavior of ISP IP blocks, etc.

It may be that this customer was specifically and exclusively targeted, but even so FB should have mechanisms in place to deal with and block this. This is not a good sign, but then as I've mentioned before I don't think ads is a viable model for FB, so perhaps their energy is being directed elsewhere.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#295

Post by wildtype »

cientifico wrote: post #201
mattmanser wrote: post #132

Earlier quoted context omitted.

Access logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions. Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for ex…

easy: (new Image()).src = document.location.href + "Hey, i have javascript And then compare in the log.

genius

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#296

Post by rapind »

pmarsh wrote: post #248
rapind wrote: post #239

Earlier quoted context omitted.

Couldn't agree more. It should be almost trivial for them to track down bots and flag bad accounts. I definitely wouldn't classify it as one of their harder problems.

Not saying that's not the case, but you have no data or knowledge to back any of that up. I wouldn't assume a problem is easy or hard until I've got sufficient info on it. Too often I've heard "how could XYZ not have done this, fixed that?" Then have that same party sign on board to fix this "easy" problem and get themselves in a world of hurt.

Well I do have at least some data to back that up. I know that they are a walled garden that keeps a ton of user data for every account (even the subset represented in Open Graph is significant). And it's with this knowledge that I make my conclusion.

* Accounts subsisting of an unusual proportion of ad clicking activity compared to their other activities are likely bots.

* New accounts with no history but high ad clicks are likely bots.

Before a click is counted, check the source account for it's human rank (activity-to-clicks). If it's below a certain threshold (tuned over time) than don't bill your customer for it.

The algo's to determine human rank can start fairly simple and become more complicated and accurate over time.

In terms of their current sophistication I would definitely call this an easy problem from a tech standpoint. Scaling to millions of concurrent users? Hard. Attracting and keeping millions of users? Hard.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#297

Post by rscale »

tripzilch wrote: post #264
rscale wrote: post #222

Earlier quoted context omitted.

One as-yet-unmentioned technique is to adjust a bot score by taking an OS fingerprint, and comparing that to the listed user agent. It's not perfect for a variety of reasons, but I found it to be a useful input for a similar bot detection problem. That said, this was a long time ago so I'd need to re-run some experiments to see if the hypothesis remained valid. For those not familiar with OS fingerprinting, it's a me…

On the packet level ... hm that's pretty clever, definitely not trivial to fake, either. Just that if your bot runs from, say, Windows 7 and it spoofs an IE8 user-agent header, or even runs by directly automating IE itself, how do you detect it then? Both of those scenarios are not unlikely at all.

If the bot spoofs a user-agent that is viable for the OS, then you need to rely on other signals. It only works as a component of a defense in depth strategy.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#298

Post by joeblau »

This is a pretty interesting finding. Has your team considered acutally spinning off a consulting branch or creating a plugin that other companies can use to determin if they are paying for bot clicks? That would probably be pretty helpful for others and a great way to draw attention to the issue. I'm sure if they see a growing number of companies stop advertising on the site, they may start to pay attention.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#299

Post by tlb »

Two other hypotheses they don't seem to have ruled out:

- Their site is so awful that 80% of people click back before the page finishes loading, thus appearing to not have javascript

- Their ad is so awful that almost no people click it and a few bots make up 80% because the total is so low.

Re: Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

#300

Post by SwellJoe »

tripzilch wrote: post #275
SwellJoe wrote: post #231

Earlier quoted context omitted.

So, this is why people don't take you seriously. I know a couple of guys like you in my local activist community, who take a very hostile "I know the truth and you're all fools!" attitude, despite the fact that their audience is mostly very sympathetic to most of their assertions. We know spying on the Internet takes place; HN is full of cantankerous old Internet geeks who've seen, first-hand, plenty of examples of t…

Hey, calling people "schizophrenic" is not cool. Just sayin'.

You're right. It was an unproductive method of describing the behavior I was seeing (and was too easy to interpret as saying the person is schizophrenic rather than exhibiting behavior I associate with schizophrenia). It was also insensitive to schizophrenics.

In my defense, schizophrenia runs in my family, and I'm very familiar with it...I don't think of it as an insult. But that's a local custom in my family that I shouldn't think follows in the rest of the world.

Post reply on HN