Should I worry if website is being scrapped first month of launch?

#1

Post by nextburb » Wed, Feb 03, 2021, 8:26 PM UTC

We are early stage startup (Nextburb) pivoted from B2B to B2C due to pandemic late last year. We opened up most of our data and pages iteratively to ensure google indexes sooner and faster. But it is slow and steady as we believe Google is trying to learn engagements based on session duration and bounce rate.

All of a sudden we are however seeing some bot traffic coming in from portland area with 200+ user all from same location and hitting many of our pages but nil session duration. We are worried about this and hope this doesn't get penalized for our organic traffic / engagement. Should we do something about it or ignore since there are many more productive things for us to do for our users.

Thought to get input from hackers here! Thanks in advance.

Re: Should I worry if website is being scrapped first month of launch?

#2

Post by SavageBeast » Wed, Feb 03, 2021, 8:54 PM UTC

It sucks that as soon as somebody builds something cool, along comes some jack ass looking to exploit that persons cool thing for no money or benefit to the creator (usually to the creators detriment).

Some simple things you can do:

Look at your access logs - where google is coming from (crawling) will be pretty evident - note that IP address and be sure you don't block it!

Next, read up on iptables in bash and specifically blocking a range of IP addresses or an entire netblock.

Then using your logs, block a range of IPs this scraper is making use of. Keep doing this until your scraper runs out of IPs.

If its an AMZN cloud IP range or a GCP range - you know no customer with a browser is coming in from there - BLOCK!!

Every time you do this use google webmaster tools to request a re-crawl so you're fairly certain you didn't block GoogleBot.

This may take care of your problem depending on the sophistication of your scraper.

This isn't perfect of course (I know... I know ... no need to school me here) but its cheap and you can do it right now quickly. If for no other reason than to frustrate your scraper.

Thats the ultra quick and dirty cheap way to handle it. Also, free advice is generally worth every dollar you pay for it.

Re: Should I worry if website is being scrapped first month of launch?

#3

Post by nextburb » Thu, Feb 04, 2021, 10:46 AM UTC

SavageBeast wrote: post #2

It sucks that as soon as somebody builds something cool, along comes some jack ass looking to exploit that persons cool thing for no money or benefit to the creator (usually to the creators detriment). Some simple things you can do: Look at your access logs - where google is coming from (crawling) will be pretty evident - note that IP address and be sure you don't block it! Next, read up on iptables in bash and speci…

Thanks for the suggestion.bandwidth is always limited and don’t want to run myself and team for these things but something are pretty straightforward and can be done to block.