61–70 of 85 posts
Basic question before I can recommend this to my boss: can it scrape G2? (or any other page behind CF)
""" What happens if my scraping fails? Not to worry! We will make every effort to determine the cause of the problem and assist you in resolving any issues with your scraper.
Additionally, please note that unsuccessful scrapings will not be included in your monthly quota. """
I'm curious the feedback mechanism for failed scrapes. Is there any validation configuration or an email notification I can configure in the event the target changes their page layout or DOM or whatever happens to cause interference.
I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…
for an (unlimited) free local option, https://webscraper.io/ may do what you want. It is simpler than this one (no proxy/scheduling/API...) but the scraping rules are quite elaborate.
If running the model was cheaper I would even say run the whole page through ChatGPT and ask it to format the information on the page for you.
I scraped https://bot.incolumitas.com/ . Results do not look good, sorry! { "new_tests": "{\n \"puppeteerEvaluationScript\": \"OK\",\n \"webdriverPresent\": \"FAIL\",\n \"connectionRTT\": \"FAIL\",\n \"overrideTest\": \"OK\",\n \"puppeteerExtraStealthUsed\": \"OK\",\n \"inconsistentServiceWorkerNavigatorPropery\": \"OK\",\n \"inconsistentWebWorkerNavigatorPropery\": \"OK\"\n}", "detection_tests": "{\n \"intoli\": {\n…
and ip range of "us" geo proxy gives is_abuse: true. Consider this to be okayish though, given that this is a default proxy pool.
This marketing bit seems a bit conflicting: "With MrScraper, you won't be blocked. We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio." "won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible…
One would hope that anti-blocking measures are implemented ethically and the documentation clarified to reflect that.
Your assumption that blocking is somehow ethical by default is not unproblematic.
There's a world wide web built by academics for free exchange of information and there's a closed garden web built by major capitalists.
Just how free that exchange of information should be is not a settled problem. Some very libertarians argue along the lines of information "wanting to be free". Some commercial entities seem to identify copyright and trademark law with moral doctrines. There are plenty of arguments for in-between positions as well.
If we look at less democratic societies, the efforts made to circumnavigate state censorship are publicly lauded as morally good actions by the international community. Could an analogy be drawn to large corporations censoring the less fortunate in a economically uneven societies too, for instance?
Earlier quoted context omitted.
One would hope that anti-blocking measures are implemented ethically and the documentation clarified to reflect that.
> anti-blocking measures are implemented ethically Your assumption that blocking is somehow ethical by default is not unproblematic. There's a world wide web built by academics for free exchange of information and there's a closed garden web built by major capitalists. Just how free that exchange of information should be is not a settled problem. Some very libertarians argue along the lines of information "wanting to…
Your assumption that blocking is somehow ethical by default is not unproblematic.
is itself an assumption.
The problem I'm concerned with is aggressive (either deliberately or ignorantly) crawling/scraping of non-commercial sites which often lack the financial resources to defend against activities enabled without apparent concern by tools like the site here.
If a site allows reasonable access in good faith, then subverting those limits and constraints for self-serving reasons is ethically dubious at best, and any service not addressing that while promising to enable that subversion should be questioned.