Live data from Hacker News

The Guardian Is Being Swamped with 'Dark Traffic'

uk.businessinsider.com

61–70 of 70 posts

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#61

Earlier quoted context omitted.

That's the beauty of hackernews, it doesn't need a paywall or advertising. The presence of the users is the payment.

But hackernews links to free information sources. No free information sources => no hackernews...

Plenty of the information sources linked to are free of advertising and not behind paywalls. In fact, those are probably the better information sources.

That's how the web started, remember: no ads. Just free sources of information.

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#62

Earlier quoted context omitted.

Nominated for the most classy comment of the month. That the product manager of The Guardian would take the time out from his (no doubt) busy day to help a user to block tracking is an amazing display of trust in that the user knows best what is good for them. Thank you.

It's more like he knows that helping one single user like this isn't going to make a dent in the tracking they do on virtually all other users. Now, if they voluntarily stopped tracking all or a significant portion of their users, I would be shocked. Of course, that isn't going to happen.

Of course there is always a way to put a negative slant on just about anything.

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#63
post #3

I built my personal extension which blocks all the traffic from Dan Pollock's list [1], then blocks all the traffic from major service providers (google analytics, etc) and social networks (fb, tw, google, etc) when not on their website. Referral and user agent headers removed, I haven't found the need to remove other headers. Currently working on preventing (and manually allowing) all xhr/script/image requests 2 sec…

I really wonder how this 10KLines is handled by the network subsystem. Is it compiled into a big regexp ? or another form of compacted runnable logic ? or at which point does it slow local name lookup.

At a guess, by doing a match first on a substring using a hashtable. That way if something is a 'candidate' you can hit a more expensive datastructure to figure out if you really have a hit without burning a lot of cycles.

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#64
post #41

I think it's clear that almost all of the dark traffic is simply https sites. It's no necessarily apps, it can just be gmail... Someone really should make it a default to pass the referrer even for https.

Haha, wat?

edit (less obtuse): there should be less passing of referral headers, not more. Browsing is already such a leaky experience privacy-wise, we shouldn't be clamouring for it to become worse...

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#65

Earlier quoted context omitted.

I really wonder how this 10KLines is handled by the network subsystem. Is it compiled into a big regexp ? or another form of compacted runnable logic ? or at which point does it slow local name lookup.

At a guess, by doing a match first on a substring using a hashtable. That way if something is a 'candidate' you can hit a more expensive datastructure to figure out if you really have a hit without burning a lot of cycles.

This kind of long blacklists seems so fitting for Tries. I should grep *nix kernels (I wish I had windows >Xp source too) I guess.

Linux (well, glibc, from http://unix.stackexchange.com/questions/81979/how-does-etc-h...):

http://repo.or.cz/w/glibc.git/blob/HEAD:/nss/nss_files/files...

Windows :

#tbd

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#66
post #14

Earlier quoted context omitted.

Looks like that list hasn't been updated since our domain switch. If you really want to block our internal analytics (which are in practice fairly harmless) replace "hits.guardian.co.uk" with "hits.theguardian.com".

I wonder why companies who host their own analytics choose to use a separate request to track users? Surely it is better just to get the data from your own content web server logs ? Wouldn't you get the same information while saving on the extra HTTP request? It would make the site loading times slightly faster as well.

Server logs will track all requests from crawlers, accelerators, and cancelled navigations. An AJAX callback will only happen for real human visitors that actually view the page.

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#67

Earlier quoted context omitted.

But hackernews links to free information sources. No free information sources => no hackernews...

Plenty of the information sources linked to are free of advertising and not behind paywalls. In fact, those are probably the better information sources. That's how the web started, remember: no ads. Just free sources of information.

Not behind paywalls, I agree. Free of advertising... it would be interesting to have stats at hand, but I'm not so sure.

Anyway, thank you for reminding me I'm old enough to remember how the web started :D

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#68
post #49

" not knowing that data hurts the Guardian's ability to serve those readers relevant content." Actually one of the main reasons why I use various anonymizers is that I don't want relevant content for the same reason I don't want to see Facebooks's "top stories" -- most often it turns out to be totally irrelevant, clickbait or complete bullshit. Leave me the choice to what's interesting for me and what I want to see.

But how are they (FB, Google, guardian) going to make money if you wont let them tell you what to think????

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#69

Earlier quoted context omitted.

> The Guardian is funded by advertising and this limits their ad sales story ... inevitable downvotes from the privacy crowd. The privacy crowd are right; and so are you. This is a huge internal conflict the web today - how do you make it pay, keep it free and not have it track users?

This presumes that 'keep it free' is actually desirable. Free journalism means the person reading it is the product. Perhaps the less of that, the better.

the existing model has been to subsidize newspaper sales with advertising for a very long time. This same model carried forward through radio and television.

The alternative is a pay-per-view service, or subscribing to wires. I haven't done any research into the viability of that type of service, but it is a paradigm shift.

Re: The Guardian Is Being Swamped with 'Dark Traffic'

#70
"The frustration here is that search, apps and HTTPS traffic all represent different types of readers arriving at The Guardian for different reasons — and not knowing that data hurts the Guardian's ability to serve those readers relevant content."

I think they meant to say 'relevant advertising' there not 'relevant content' as the content should, in theory, be the same regardless of how you got there. The interesting bit is that I've seen advertising contracts where you can't advertise with unapproved networks on a referred link from a Google SERP. Only on the second click can you do that pop-under or egregious flying frisbee ad. So if you are trying to be 'safe' you don't do any of that nonsense if you can't tell the difference, and I'm guessing that cuts into revenue.

Post reply on HN