Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Dealing with a competitor who is scraping my content and ranking higher

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 130 posts

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#121

Post by setr »

jklein11 wrote: post #101
jordan801 wrote: post #91

Earlier quoted context omitted.

Anything receiving 5k visits a month, is likely crawled at least once a day unless it's told not to. In this case, Google could check crawls and determine the original source of content between the two competitors.

If the source that is copying is also being crawled once per day isn't possible that it would be crawled first and appear to Google that it is the original source?

presumably site A got popular before site B started scraping it; else there wouldn't be much point in scraping A. There is ofc the scenario was A was a generally unknown resource, and B realized it had the potential to be valuable if you just SEO'd it right.. but there's still probably evidence somewhere that A cropped up first (how did B stumble into it?)

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#123

Post by rakjosh »

laurieg wrote: post #12

If you are able to detect them, serve them garbage data instead of blocking them. Mix in some false entries etc. If you completely block them they will work hard to work around the block, but if you give them some bad data randomly it will take them longer to notice. Also, you can use the fake data as evidence they scraped you.

That is what I did initially. My API was sending garbage data Saying this website is shutting down, please visit [mysite] for more info :D. But later on, they figured, I was sending garbage data and people were visiting my site because of that. I even put in a tracked link to see how many visitors I'd get from there site and I was getting around 500 people every day.

It was like a cat and mice game for a few weeks.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#124

Post by rakjosh »

timrichard wrote: post #64

> If I change certain things, they counter it immediately and make it work. If the content is markup based, are your countermeasures about changing the IDs, classes, or overall tag structure of the markup you serve? I was wondering if you could have several variations of the above, and serve your content via a random one each time that would be visually indistinguishable to a human viewer. The person maintaining the…

The content is served using API that is open for everyone to use. And that is making it difficult for me to protect it. I've tried changing the structure of API response several times, but they counter it within few hours. Tried adding unique headers to requests etc (That worked for quite some time) but they figured it out ultimately. I come up with some solution, they figure it out in a day or two. And that went on for few weeks

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#125

Post by bausshf »

bena wrote: post #109
codedokode wrote: post #100

Earlier quoted context omitted.

Copyright protects only creative works, not any information as I understand.

It's a little weird. Trap streets often aren't protected by copyright as the existence or non-existence of something is a fact and facts aren't copyrightable. And maps are nothing more than a collection of facts. As are recipes. You can take the recipes in a cookbook, write them down, and disseminate them how you choose and no one could say boo. Which is why Coca-Cola guards its recipe carefully. Once it is known, th…

Include a base64 encoded image as a street name.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#127

Post by DEADBEEFC0FFEE »

mullen wrote: post #117

I worked at a company that had this problem and while I was not involved with the solution, I did sit near the guy that was and had extensive talk with him on how to resolve the issue (Or solve the issue well enough). * Require accounts with good robot account creation detection. Actually, not hard since there is a lot of canned coded for this. * Subscribe to Tor and Proxy IP collection services and then block all of…

Banning Tor and proxies is pretty unfair to legitimate users who use those services.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#128

Post by tehlike »

DEADBEEFC0FFEE wrote: post #127
mullen wrote: post #117

I worked at a company that had this problem and while I was not involved with the solution, I did sit near the guy that was and had extensive talk with him on how to resolve the issue (Or solve the issue well enough). * Require accounts with good robot account creation detection. Actually, not hard since there is a lot of canned coded for this. * Subscribe to Tor and Proxy IP collection services and then block all of…

Banning Tor and proxies is pretty unfair to legitimate users who use those services.

While i agree, the op isnt doing a charity either, and op needs to keep their interest at a higher priority to be able to keep providing the service.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#129

Post by ddgflorida »

JonoBB wrote: post #80
slededit wrote: post #11

This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en

I had my entire marketing site copied, word-for-word. I still have no idea why (other than perhaps to damage google ranking due to duplicate content?). I reported it to google, and they removed it from search results in a few days. I also reported it to the site's host, who took it down completely.

Almost the same here - they copied my whole website, improved the user interface, and added a little content. Luckily I still outrank them :)

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#130

Post by rakjosh »

This is my website if anyone was wondering: https://freephonenum.com/

My competitor was scrapping all my phone numbers and SMS received from my site and displaying it on their site. It took me several months to figure that out. I was seeing abnormal bandwidth usage for JSON (my API endpoint) but never realized that someone was constantly pinging my APIs.

[edit] Now, I've removed the API and everything is served as HTML (not easier for someone to scrap my API)

Post reply on HN