Live data from Hacker News

Ask HN: Dealing with a competitor who is scraping my content and ranking higher

news.ycombinator.com

71–80 of 130 posts

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#71
If you're able to detect when your content is being scraped, you can send them the same text but with "confusable" [1] characters (i.e. utf8 characters that look similar to ascii characters).

[1] https://unicode.org/cldr/utility/confusables.jsp?a=test&r=No...

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#72

> If I change certain things, they counter it immediately and make it work. If the content is markup based, are your countermeasures about changing the IDs, classes, or overall tag structure of the markup you serve? I was wondering if you could have several variations of the above, and serve your content via a random one each time that would be visually indistinguishable to a human viewer. The person maintaining the…

Easily bypassed: just retry on failure until scraper gets syntax it likes.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#73
post #67

Add sensible rate-limits (modelled after an average user's expected usage) and present a captcha after that. Make it per-IP. Sure, the crawler can change IPs, but I doubt they have unlimited addresses and eventually they will run out. Another option (if you can reliably detect the scraper) is to poison their data by sending them bad (but valid-looking) data.

This will be more effort to implement than it will be to route around: between free VPNs and TOR, everyone has infinite IPs.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#74

This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en

Yes, I would combine this with a couple of fake entries, and when those show up on their site, you file DMCA claims with google, and with their hosting provider. An american company has no choice but to shut their site down under threat of DMCA claims. If the site gets back up, add another fake entry, and do it again when they scrape you, until they give up.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#75
post #40

Earlier quoted context omitted.

Please don't invoke the CFAA for scraping.

Why not? Scraping seems like exactly what the law is aiming to prevent, in spirit and in letter.

The scraper is 100% authorized to access the data. It's the rehosting that is the problem.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#76
post #32

Earlier quoted context omitted.

The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.

ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection. An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probabl…

this won't work in this case. The site is being specifically targetted. A custom captcha will be trivial to bypass. The reason it worked for you would be your site was just bulk scraped and no patterns could be found in your captcha to match it to a solving solution so the spammer moved on to the next.

When your site is specifically targetted, captchas will not work

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#77
post #38

Earlier quoted context omitted.

They're probably using https://2captcha.com/ They pay 50c per 2 hours to the workers, and API access costs 50c-$1 per 2000 solved captchas. I think most of the people doing this are in India, where US 50c for a couple hours translates well enough. (Completely naive about the truth of the situation though.)

One of the regular spam messages mentions the software they're using ("You read this, so you know it works!"), so it's pretty clear that ReCaptcha has already been defeated. I'd mention the software, but I'd rather not give them publicity. Others have been able to beat reCaptcha too, though: https://www.youtube.com/watch?v=fsF7enQY8uI

didn't watch, but the Russian xrumer forum spammer has been at the forefront of captcha solving for the last decade or more.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#78

> If I change certain things, they counter it immediately and make it work. If the content is markup based, are your countermeasures about changing the IDs, classes, or overall tag structure of the markup you serve? I was wondering if you could have several variations of the above, and serve your content via a random one each time that would be visually indistinguishable to a human viewer. The person maintaining the…

Since the seemingly colluded disappearance of any freely available finance data APIs (excepting IEX), both Yahoo and Google Finance employ this method of randomizing class names on pages with finance data (quotes, fiscal data etc.). Inspect element on those pages for a good example of this tactic. I feel like this could make it much more difficult for your content to be stolen.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#79
post #20

Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if…

I don’t disagree with the content of your post and I know this is a US cultural trope at this point, but trivialising rape benefits precisely no-one.

"Federal pound me in the ass prison" is a quote from the movie Office Space.

If you already knew this, then I guess the applicable movie quote this time is "Lighten up, Francis" from Stripes.

Re: Ask HN: Dealing with a competitor who is scraping my content and ranking higher

#80

This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted. https://support.google.com/legal/answer/3110420?hl=en

I had my entire marketing site copied, word-for-word. I still have no idea why (other than perhaps to damage google ranking due to duplicate content?).

I reported it to google, and they removed it from search results in a few days. I also reported it to the site's host, who took it down completely.

Post reply on HN