Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: MrScraper – A visual web-scraping tool

mrscraper.com

Post reply on HN
Print view View original

21–30 of 85 posts

Re: Show HN: MrScraper – A visual web-scraping tool

#21

Post by buffer_overflow »

bdcravens wrote: post #2

This marketing bit seems a bit conflicting: "With MrScraper, you won't be blocked. We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio." "won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible…

Thanks for sharing your point of view!

I will rewrite the copy to make better statements. Thank you so much

Re: Show HN: MrScraper – A visual web-scraping tool

#22

Post by buffer_overflow »

jonatron wrote: post #14

I scraped https://bot.incolumitas.com/ . Results do not look good, sorry! { "new_tests": "{\n \"puppeteerEvaluationScript\": \"OK\",\n \"webdriverPresent\": \"FAIL\",\n \"connectionRTT\": \"FAIL\",\n \"overrideTest\": \"OK\",\n \"puppeteerExtraStealthUsed\": \"OK\",\n \"inconsistentServiceWorkerNavigatorPropery\": \"OK\",\n \"inconsistentWebWorkerNavigatorPropery\": \"OK\"\n}", "detection_tests": "{\n \"intoli\": {\n…

Thanks for reporting. I'll review this!

Re: Show HN: MrScraper – A visual web-scraping tool

#23

Post by buffer_overflow »

jerriep wrote: post #12

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…

Hi there,

I'm currently working on standard pagination (click next page button) and click button + infinite scroll.

What you comment is not currently possible with a single scraper, you would need to send one to collect links and then scrape those links. But I'm also working on "nesting data" feature, and what you comment should be possible in an ETA 2-3 weeks max.

Thanks for commenting!

Re: Show HN: MrScraper – A visual web-scraping tool

#24

Post by pocket_cheese »

idjdhdhddjaj wrote: post #17
jerriep wrote: post #12

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…

Try browserflow.

I 100% recommend browserflow. It's fucking awesome!

Re: Show HN: MrScraper – A visual web-scraping tool

#26

Post by jefozabuss »

bdcravens wrote: post #2

This marketing bit seems a bit conflicting: "With MrScraper, you won't be blocked. We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio." "won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible…

"It won't be blocked" = they imported the stealth plugin most likely

Re: Show HN: MrScraper – A visual web-scraping tool

#28

Post by jerriep »

idjdhdhddjaj wrote: post #17
jerriep wrote: post #12

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…

Try browserflow.

Ooh yeah, works great, thanks! It's a pity I have to buy a subscription as my needs are more of a once-off.

Re: Show HN: MrScraper – A visual web-scraping tool

#29

Post by factsaresacred »

jerriep wrote: post #12

I've had a look at a number of these "simple" (i.e ones where I don't have to write a complex script) scraping tools recently and none of them seem to support what I consider to be a fairly common scenario of navigating to sub pages. In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item…

I had a nice experience with https://simplescraper.io for a similar use-case. Was able to scrape a few thousand URLs without too much fuss.

The biggest complication with visual scrapers is all the edge cases. The selector algorithms usually become a mess on any complex website especially if there's uneven data.

Then you have css selectors no longer working and so on. Very brittle.

Re: Show HN: MrScraper – A visual web-scraping tool

#30

Post by brap »

toxicFork wrote: post #27
brap wrote: post #25

Just how many web scraping tools do we need? It seems like every month there's a new web scraper on HN, is that really such a common task that we need dozens of tools?

If you don't need it don't use it.

I wasn't going to, still I think it's a fair question.
Post reply on HN