Live data from Hacker News

Ask HN: Dealing with a competitor who is scraping my content and ranking higher

news.ycombinator.com

31–40 of 130 posts

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#31
Something I haven't seen mentioned yet, but did you try to simple get in contact with them to discuss the issue?

Since your website is free, it might be worth combining forces to serve your users better. I know it's hard to swallow, but in the hand what matters is that what you do is useful to people isn't it? And if your users are moving away it probably is because `some` of the things they do is right?

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#32
post #9

Google ReCaptcha is a pretty low friction way for (most) users to get your content. You can do this while serving your real content to Google.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection.

An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probably less annoying to your visitors.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#36

This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en

This is an excellent idea, especially because the system is automated and allows anything and everything through.

Just submit whatever politically incorrect thing you like in with a bunch of P2P links and watch the dissent disappear :)

Instructions and discussion here: https://news.ycombinator.com/item?id=17787302

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#37
When you say "my content", are you referring to content that you created, like a blog or online resource or are you referring to content your users created? Also is the information they're copying factual in nature (like a map or results of a calculation)?

If it's factual in nature: It's not protected by copyright.

If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown notice. Your best bet is really going to be making yourself unscrapable.

If it's content you created: File a DMCA takedown notice.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#38
post #9

Google ReCaptcha is a pretty low friction way for (most) users to get your content. You can do this while serving your real content to Google.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

They're probably using https://2captcha.com/

They pay 50c per 2 hours to the workers, and API access costs 50c-$1 per 2000 solved captchas.

I think most of the people doing this are in India, where US 50c for a couple hours translates well enough. (Completely naive about the truth of the situation though.)

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#40
post #20

Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if…

Please don't invoke the CFAA for scraping.

Why not? Scraping seems like exactly what the law is aiming to prevent, in spirit and in letter.
Post reply on HN