Viewing profile — RobSm

You can always stop bots. Add login/password. But people want their content to be accessible to as large audience as possible, but at the same time they don't want that data to be …

The reason is different. When you need to scan 5 million IPs on 22 port is one thing, but 5 million IPs times 100 or 1000, then you simply run out of resources and since 99% of peo…

There are at least 1 of us! Looks weird :)

You need to upgrade your knowledge. Use only [2-9]. That's it.

Google scrapes everyones data yet you use Google every day and find it useful.

No

People don't care about such content and so does Google.

Explain more about it

And if I open the website in my browser and then copy from browser to my computer, then all is good?

So in the scale of google, 'not many' would be some few million per month? And all is good then, right? Even you use their scrapped data probably daily and are totally fine with th…

And if you manually copy someone's data they worked hard to generate to go and resell, then it's ethical?

This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have…

No. robots.txt is not something that is defined and enforced by the law. Just because someone came up with some 'recommendation' like robots.txt does not mean this is the law

If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people s…

How many contracts google breaches scraping billions of pages every month?

How one goes about using TLS proxy? Are there any services? You have any links? Thanks.

What's the point of TLS proxy?

"and offered it to everyone for free, making a fortune" - this makes no sense

Thanks, go it. You can delete now

""

I get it why someone else scrapes it. But why customers upload data in the first place? Aren't they interested in getting some OTHER data from you and that OTHER data may as well b…

Makes little sense - customers upload data to you and they don't want any data back? Really?

Building API is 5 times easier than building routes for your public webpages, which is basically an 'API' as well.

If you build your site in a way that multiplies each request 10x, well then that's what you get. Don't do that and you won't have issue with requests. Or handle those requests prop…

Exactly. I am surprised that the 'devs' can't figure out a way to block only annoying/excessive scrapers. Most likely they are just lazy and then just put 3rd party 'solution' and …