Live data from Hacker News

Show HN: CloudScrape – Cloud-based web scraping platform

cloudscrape.com

11–20 of 109 posts

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#11
Nice timing, as just this week I had my first exposure to the Cloudscrape platform. Credit where credit is due, I found the platform refreshingly feature rich even at lower tier (i.e. free) account levels compared to other offerings from competitors.

Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all the corpses littered along this path.

Recommendations:

Define solid, real-world examples of your platform in practice and really, no bullshit, what makes you different from every other platform that's 97% polished out there?

You didn't meet my needs because I needed much more in the way of parsing network requests from pages, mostly in handling if/then scenarios. A relatively simple scenario, to ME, and there-in lies the challenge of the crawler market, it's the last 5% that's custom to each scenario that so often delivers 98& of the value.

Suggestion: Identify and use the talent you obviously have to destroy some high value markets. Hint: Prioritize targets that still champion a desktop app at any point in the top 5 in terms of data collection.

Outside of that and far more niche (and subsequently far lower priority, allow for parsing / handling handling network requests and responses via custom app/API instead of just block all, some, or none)

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#13
Hit a certain threshold and you'll most likely get IP banned by the site you're scraping or heavily throttled unless the site does not care to minimize bot traffic (which can cost the site owner valuable bandwidth and server resources)

You may also get sued...

If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bots.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#14

I need to add their IP range to my blocklist.

If these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#16
Bot detection tip:

Humans don't read pages at dozens per minute.

Not hard to code for that, especially if your static content like images are served by alternate servers, so you can just focus on your content servers.

The problem I am dealing with lately is massive bot farms where you do not see a repeating IP for an hour. I catch them eventually but it takes a toll.

But this is why you preemptively block all of aws and servers like them.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#17
post #7

Earlier quoted context omitted.

Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.

It might be desirable to make data available to humans, but not bots, no?

OK, how about hiring hordes of cheap human scrapers from Mechanical Turk? Everything visible can and will be scraped if need arises, the only variable is price.
Post reply on HN