Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advertisers scramble as ‘non-human traffic’ eats up online budgets

theglobeandmail.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#81

Post by aruggirello »

I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a "ticket ID" so the (very few) real users caught can still contact the website admins to be unblocked.

The (almost feature-complete) version running at the site above was able to automatically identify and successfully block a number of bots found scraping (or, attempting to scrape) the site since 2012. Though it wasn't an easy task, it has been quite successful at blocking most of them since their second (often first) request.

In fact since the beginning, and largely upto now, most bots and botnets can still be easily spotted. Some by their user-agent, some falling into bot traps ("honeypots") which are invisible to regular users, some failing to load images (something that a regular user wouldn't do), some crawling the site at an unreasonable pace... most usually don't provide an http referer, some even appear to have a number of cookies set for your domain.

It's not always easy to spot them though - some botnets can only be undone by blocking their full IP subnet, or permanently banning their (sub)domains, like .cn, .ua or .br (if you don't have legitimate traffic from those domains).

Unfortunately, I haven't had enough spare time to complete my project (which tests all of the above conditions to spot bots, but is not ready for a release), but I could if there is some market - or I could sell the project if somebody is interested.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#82

Post by ddebernardy »

sambrand wrote: post #76
eli wrote: post #37

Earlier quoted context omitted.

CPA simply doesn't make sense for all ad campaigns. When Lexus buys ads on a financial news site it's for branding -- they want to get their logo in front of their target demographic. The ads aren't optimized for clicks or leads -- sometimes there isn't even a clear call to action.

Also with CPA, if I'm a seller, I'm not going to sell to you unless I know your landing page converts. That's asking me to do a lot of due diligence or assume a lot of risk.

There's also the problem of being sure your conversions are attributed to you.

But these aren't that much of a problem in practice, really. See how affiliate marketing works: it's CPA for all intents and purpose masquerading under a different name. The biggest difference might be that the vendor will contact you, rather than make some kind of bid, if he thinks you've the right kind of traffic.

If your audience is interested in a particular product, you quickly discover it through common sense and trial and error, as you can normally see the click-through rates.

Also, a big ticket item with a low conversion rate sometimes earns better than a smaller one with a high conversion rate. As such, maxing out your ROI is really a matter of seeing what contributes to your bottomline.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#83

Post by jacquesm »

aruggirello wrote: post #81

I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a "ticket ID" so the (very few) real users caught can still contact the website admins to be unblocked. The (almost feature-complete) version running at the site above was able to autom…

It's not hard to make a 70% effective bot blocker. It's extremely hard to make a 95%+ effective bot blocker without having a large number of false positives.

It is (probably) impossible to make a 99%+ effective bot blocker without having a very large number of false positives.

I've been working on this (and abusive user blocking) for a long time now and I'm roughly where you are, I have a product, it could be launched but I'm very wary of launching a half baked product that would eventually end up being disabled. I've had the help of a friendly corporate entity that gave me millions of datapoints to play with allowing me to look into their kitchen both on the support end and on the incoming traffic end as well as verified abusers.

This identified various fraud such as ad click fraud, affiliate scams, repeated harassment by the same users under different names, spam of all kinds and so on. It's been working there for more than 4 months on a fairly high level of confidence. And yet, I'm still wary of rolling it out on a larger scale.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#84

Post by jacquesm »

pessimizer wrote: post #21
arbitrage wrote: post #5

Earlier quoted context omitted.

William Gibson could probably give you some ideas on what that says about us. He started writing about this fictionally decades ago.

Or Frederick Pohl and Cyril Kornbluth https://en.wikipedia.org/wiki/The_Space_Merchants (Or Alvin Lucier: https://en.wikipedia.org/wiki/I_Am_Sitting_in_a_Room )

> https://en.wikipedia.org/wiki/The_Space_Merchants

That's one of the best SF books ever.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#85

Post by kefka »

chadgeidel wrote: post #52
goshx wrote: post #17

Besides the non-human traffic, most online ads are created to not be seen. I was even thinking about creating a blog post to call out the marketers out there to change this, but I am too lazy... specially to make them make more money. I am so used to how ads that cover the entire page work that I manage to pass by them without even knowing which brand was advertising there. Ads that force you to watch a video for X s…

I will never understand the YouTube ads that show 3-4 seconds of black screen and then "allow" (not sure what controls this) the "skip this ad" after 5 seconds. You (the advertiser) have to capture my interest in the first 10-20 frames of video, and pitch me in 5 seconds. Anything else is a waste of your money.

And YouTube only has one chance as well. He fact I'm seeing this annoying video ad means I have already installed Ad block edge and haven't reloaded the page.

Your move.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#86

Post by BillyParadise »

justizin wrote: post #42
duaneb wrote: post #26

Earlier quoted context omitted.

Does the average person like to purchase an item they already own if they see an advertisement for it? Of course not.

Campaigns like this, while frustrating for existing users, aren't there to target us, they're to target someone who filled out the sales form halfway and got distracted.

If I was ever going to buy from New Relic, I wouldn't now - I've seen their retarget ads at least a thousand times. If I didn't care about click fraud, I would have made a point to make sure I cost them a few dollars.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#87

Post by eli »

hyperpape wrote: post #61
eli wrote: post #37

Earlier quoted context omitted.

CPA simply doesn't make sense for all ad campaigns. When Lexus buys ads on a financial news site it's for branding -- they want to get their logo in front of their target demographic. The ads aren't optimized for clicks or leads -- sometimes there isn't even a clear call to action.

Yup, but then why bother with CPM? Just buy display ads for a fixed price in publications they believe their customers might read. Sure, it's not an improvement over what they did in the 50s but they don't have to worry about fraud.

Yes, that's one of the primary ways my startup sells ad space :)

It also means we have time to really look at each ad to make sure it's a fit for the audience or if there are ways to make it better.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#88

Post by eli »

brandnewlow wrote: post #71
hyperpape wrote: post #61

Earlier quoted context omitted.

Yup, but then why bother with CPM? Just buy display ads for a fixed price in publications they believe their customers might read. Sure, it's not an improvement over what they did in the 50s but they don't have to worry about fraud.

Those ads are freakishly expensive. Like 10x more expensive than buying ads on FB/Google/Twitter.

My very biased opinion is that you get what you pay for.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#89

Post by GFischer »

SixSigma wrote: post #77

> Do you hate how easy it is for your computer to become slow and riddled with malware? No, I can't say I do. If such crime is so rampant, one would think it was ripe for law enforcement to do more about it. I wonder if reporting it to the police every time one visits a relative and removes crap from their computer would start a disruption.

I went to the local electronic crimes branch of law enforcement once, when my Gmail account was broken into.

They basically asked me if I was reporting large-scale financial theft, human traficking or child porn. If it wasn't one of those, they didn't have the manpower or will to pursue it.

Re: Advertisers scramble as ‘non-human traffic’ eats up online budgets

#90

Post by rebootthesystem »

jsonne wrote: post #69
rebootthesystem wrote: post #64

We stopped all advertising on Facebook. It's a mess.

Really? I've had quite a bit of success with facebook for my clients. What didn't you like about it?

Fake non-responsive non-engaging likes to pages and more. See this, it's very true:

http://www.youtube.com/watch?v=oVfHeWTKjag

Post reply on HN