Live data from Hacker News

Ask HN: Dealing with a competitor who is scraping my content and ranking higher

news.ycombinator.com

41–50 of 130 posts

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#41

Write a few things. Register them with the copyright office. Then post them and wait for them to copy it. Now you can sue them for the max and strong-arm them to take it all down or face obvious legal fees and punishments over 100k.

Can you sue for monetary damages if you yourself were not making any money off the content? If anything the other site could be called a mirror, and saving our op money on the hosting cost by diverting traffic.. (although maybe not from scraping?)

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#42
post #37

When you say "my content", are you referring to content that you created, like a blog or online resource or are you referring to content your users created? Also is the information they're copying factual in nature (like a map or results of a calculation)? If it's factual in nature: It's not protected by copyright. If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown no…

> If it's factual in nature: It's not protected by copyright

Compilations of facts can however be protected in the EU under the Database Directive.

Also, Terms of Use etc.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#43
post #32

Earlier quoted context omitted.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection. An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probabl…

> and probably less annoying to your visitors.

As someone who has used both, absolutely 100% no in every way.

Google's captcha is so superior to what you would write, please don't tell people not to.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#44
post #38

Earlier quoted context omitted.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

They're probably using https://2captcha.com/ They pay 50c per 2 hours to the workers, and API access costs 50c-$1 per 2000 solved captchas. I think most of the people doing this are in India, where US 50c for a couple hours translates well enough. (Completely naive about the truth of the situation though.)

One of the regular spam messages mentions the software they're using ("You read this, so you know it works!"), so it's pretty clear that ReCaptcha has already been defeated. I'd mention the software, but I'd rather not give them publicity.

Others have been able to beat reCaptcha too, though:

https://www.youtube.com/watch?v=fsF7enQY8uI

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#45
post #41

Write a few things. Register them with the copyright office. Then post them and wait for them to copy it. Now you can sue them for the max and strong-arm them to take it all down or face obvious legal fees and punishments over 100k.

Can you sue for monetary damages if you yourself were not making any money off the content? If anything the other site could be called a mirror, and saving our op money on the hosting cost by diverting traffic.. (although maybe not from scraping?)

I'm not sure about how this works in the US but in my country (Germany) "commercial use" does not actually require money to change hands or operating at a net profit. And last I checked copyright laws (if they apply to the content) apply to everything you create, whether you monetize it or not.

That HN comment of yours? If it's sufficiently non-trivial, it's protected by copyright. Even though neither you nor HN directly make money of it. If someone scrapes HN and republishes that comment, you could go after them and demand they delete it.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#47
Learn to accept it. Only way to stop scrapers is to make it a financial burden for them to scrape your website. There are to many ip address, proxy, vpn, and botnets for you to try and block them all. You can write code to try and tell legit traffic from scrapers traffic but then they will just figure out how to bypass that through trial and error. You can try to take legal action, but that will cost you money up front and might take longer then you think. You can try to get there service terminated, but most host won't do it without a court order. You can make you html/css hell to read randomize autogen all your class and id tags. Put all you elements in random order so the people writing the scraping code don't have a default template they can update in minutes. Use javascript to actually send the data which then you use to identify people and have a call home function so they cannot hide behind proxies and vpn and such.

These are all just things that will slow them down, but won't stop them. Make it cost them more money then they make using your data will be the only way to truly stop them.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#48

If you haven't already, try adding some "trap streets" to your data. Map makers occasionally include streets that don't exist, so if a competitors map includes it too, it's clear that the competitor copied it: https://en.wikipedia.org/wiki/Trap_street I did that with an online marketing dictionary I wrote years ago, some of the definitions included strange usage examples that contained the names of several of my frie…

Similar to trap streets are "phantom settlements", aka "paper towns", which are fake towns rather than streets.

Now, this idea is not limited to maps: Google used trap search results to catch Microsoft using Internet Explorer to scrap Google search results: https://googleblog.blogspot.com/2011/02/microsofts-bing-uses...

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#49
(Note:I am not a lawyer.) If you're a US citizen, any content that you've created yourself is -automatically- copyrighted. (That's also true in all countries where Berne Convention standards apply.) See this link for further basics: https://smallbusiness.findlaw.com/intellectual-property/what...

If they're scraping content you created, they have already broken the copyright law. (They may not realize that. Make sure they do.)

Since 1989, in Berne countries you don't even have to post copyright notice for them to be in violation. So your first step would be to notify them that they've already broken the law, and invite them to cut it out.

Your second step would be to contact a lawyer.If the other site is not in a Berne country, that might complicate things. On the other hand, they may simply not be aware that they've broken the law.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#50
post #43
post #32

Earlier quoted context omitted.

ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection. An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probabl…

> and probably less annoying to your visitors. As someone who has used both, absolutely 100% no in every way. Google's captcha is so superior to what you would write, please don't tell people not to.

How so ? It's really not that hard. I've done it, and it zeroed spam coming through a web form almost 100%.
Post reply on HN