Are you using server side rendering? There are ways to make it harder. But ultimately you don't want to sacrifice the user experience. Why don't you require registration provided by a third party like facebook or google - it's one click sign on and you can make your access requirements extremely low.
Ask HN: Dealing with a competitor who is scraping my content and ranking higher
21–30 of 130 posts
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#22This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en
He can also go after them with this. He can write them to take it down. If they don’t, go to the host and have them take the site down.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#23Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if…
That's well thought out!
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#24Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if…
Please don't invoke the CFAA for scraping.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#25Loads of great suggestions. The DMCA request is the tool you want. That will cause a downrank if you keep doing it. Problem solved.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#26If you can detect them try to serve them wrong data
Yea I was thinking a "hellban" type tactic might wreak more havoc. Just subtle enough to de-rank them.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#27If you haven't already, try adding some "trap streets" to your data. Map makers occasionally include streets that don't exist, so if a competitors map includes it too, it's clear that the competitor copied it:
https://en.wikipedia.org/wiki/Trap_street
I did that with an online marketing dictionary I wrote years ago, some of the definitions included strange usage examples that contained the names of several of my friends. When a competitor scraped us, instead of shutting them down, the boss negotiated a data licensing arrangement with the scraper instead, so we ended up getting a revenue stream & backlinks out of the incident.
If that fails, and talking to them directly doesn't work, then the DMCA is often effective. I've made DMCA requests against websites that distributed cracks of my software & they often disappeared in a couple of days.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#28Google ReCaptcha is a pretty low friction way for (most) users to get your content. You can do this while serving your real content to Google.
The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#29This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en
He can also go after them with this. He can write them to take it down. If they don’t, go to the host and have them take the site down.
"Excuse me, could you put down my TV?"
Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher
#30If you are able to detect them, serve them garbage data instead of blocking them. Mix in some false entries etc. If you completely block them they will work hard to work around the block, but if you give them some bad data randomly it will take them longer to notice. Also, you can use the fake data as evidence they scraped you.
It's not nearly as valuable as cryptographically provable evidence of the pages having identical content as yours, with yours being of an early publication date.