Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: MrScraper – A visual web-scraping tool

mrscraper.com

Post reply on HN
Print view View original

11–20 of 85 posts

Re: Show HN: MrScraper – A visual web-scraping tool

#12

Post by jerriep »

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages.

In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item and navigate to a detail page to extract further info.

Looking at your app and docs you don't seem to support this either. Is this something you are considering?

Re: Show HN: MrScraper – A visual web-scraping tool

#14

Post by jonatron »

I scraped https://bot.incolumitas.com/ . Results do not look good, sorry!

{ "new_tests": "{\n \"puppeteerEvaluationScript\": \"OK\",\n \"webdriverPresent\": \"FAIL\",\n \"connectionRTT\": \"FAIL\",\n \"overrideTest\": \"OK\",\n \"puppeteerExtraStealthUsed\": \"OK\",\n \"inconsistentServiceWorkerNavigatorPropery\": \"OK\",\n \"inconsistentWebWorkerNavigatorPropery\": \"OK\"\n}", "detection_tests": "{\n \"intoli\": {\n \"userAgent\": \"OK\",\n \"webDriver\": \"FAIL\",\n \"webDriverAdvanced\": \"FAIL\",\n \"pluginsLength\": \"FAIL\",\n \"pluginArray\": \"FAIL\",\n \"languages\": \"OK\"\n },\n \"fpscanner\": {\n \"PHANTOM_UA\": \"OK\",\n \"PHANTOM_PROPERTIES\": \"OK\",\n \"PHANTOM_ETSL\": \"OK\",\n \"PHANTOM_LANGUAGE\": \"OK\",\n \"PHANTOM_WEBSOCKET\": \"OK\",\n \"MQ_SCREEN\": \"OK\",\n \"PHANTOM_OVERFLOW\": \"OK\",\n \"PHANTOM_WINDOW_HEIGHT\": \"OK\",\n \"HEADCHR_UA\": \"OK\",\n \"WEBDRIVER\": \"FAIL\",\n \"HEADCHR_CHROME_OBJ\": \"FAIL\",\n \"HEADCHR_PERMISSIONS\": \"FAIL\",\n \"HEADCHR_PLUGINS\": \"WARN\",\n \"HEADCHR_IFRAME\": \"FAIL\",\n \"CHR_DEBUG_TOOLS\": \"OK\",\n \"SELENIUM_DRIVER\": \"OK\",\n \"CHR_BATTERY\": \"OK\",\n \"CHR_MEMORY\": \"OK\",\n \"TRANSPARENT_PIXEL\": \"OK\",\n \"SEQUENTUM\": \"OK\",\n \"VIDEO_CODECS\": \"OK\"\n }\n}" }

Re: Show HN: MrScraper – A visual web-scraping tool

#15

Post by asdadsdad »

bdcravens wrote: post #2

This marketing bit seems a bit conflicting: "With MrScraper, you won't be blocked. We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio." "won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible…

also considering tests above - \"webDriver\": \"FAIL\" - seems like you'll totally get blocked by any anti-bot

Re: Show HN: MrScraper – A visual web-scraping tool

#17

Post by idjdhdhddjaj »

jerriep wrote: post #12

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…

Try browserflow.

Re: Show HN: MrScraper – A visual web-scraping tool

#18

Post by buffer_overflow »

samanator wrote: post #5

How do you approach pages that sometimes, non deterministically, present captcha challenges? Are you using a service like 2captcha to auto-solve captchas?

Honestly, I've been working on it for two months and didn't reach this part of the roadmap yet. I was planning to use a 2captcha integration for the first approach.
Post reply on HN