21–30 of 109 posts
The tooling and intuitivity is awesome but besides that at the heart doesn't it do what any other headless javascript enabled browser does?
For example: https://phantomjscloud.com/site/pricing.html
Is there a way to get a less feature rich version of this i.e. sans auto-resolve CAPTCHA etc?
Great work none the less though.
Hit a certain threshold and you'll most likely get IP banned by the site you're scraping or heavily throttled unless the site does not care to minimize bot traffic (which can cost the site owner valuable bandwidth and server resources) You may also get sued... If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bo…
Countless sites depend on web scraping. You can scrape and be a good netizen, the two are not mutually exclusive.
I need to add their IP range to my blocklist.
Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
I then wanted to back out, of course there is no way to remove your account.
While I know what scrape/robot/etc does, I don't know how to start using this one after spent some minutes reading through on the website.
I'm wondering about use cases for this (?) Also, do they respect robots.txt? And how are they going to avoid being blocked by websites that don't want to be scraped? (I guess it would be easy to determine the IP addresses of their scrape robots).
Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.
- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical.
Is there a website where we can blacklist IP addresses of such violators ?
What exactly do you think is being violated here?
Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.
- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
That's just wrong. You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical. Is there a website where we can blacklist IP addresses of such violators ? What exactly do you think is being violated here?
If this has automated CAPTCHAs, there will be work into making much more difficult CAPTCHas. I don't know if this is sustainable. I remember 2 different sites where I havent been able to solve their captchas
Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.