Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: MrScraper – A visual web-scraping tool

mrscraper.com

Post reply on HN
Print view View original

71–80 of 85 posts

Re: Show HN: MrScraper – A visual web-scraping tool

#71

Post by mdaniel »

theFletch wrote: post #59

What about scraping PDFs on the web? Anyone have suggestions for that?

What would the expression language for that even look like, given that PDFs are basically "canvas as a service"?

I'm aware there are pdf2html toys, and sometimes they do something reasonable, but just like with web scraping the markup of the target matters a lot and so, too, would the "markup" of the target PDF

Further, just like often it is better to go after the underlying XHR instead of trying to de-React the HTML, I'll offer that when possible it would be far better to try and identify the upstream source of the information in the PDF than trying to reverse engineer a postscript VM

Re: Show HN: MrScraper – A visual web-scraping tool

#73

Post by jamestimmins »

bdcravens wrote: post #60
jamestimmins wrote: post #56

Earlier quoted context omitted.

What is a stealth plugin?

Additions to libraries like Puppeteer that help ensure that the browser being used looks more "organic", often by returning fake data that a normal browser would have (browsers have APIs with things like plugins and fonts installed etc)

Cool thanks for explaining!

Re: Show HN: MrScraper – A visual web-scraping tool

#74

Post by kmoser »

buffer_overflow wrote: post #51
somsak2 wrote: post #49

I signed up but on the setup wizard ( https://app.mrscraper.com/onboarding ) I can't seem to edit any of the input boxes ("Give your scraper a name", "Enter the URLs you want to scrape"). I'm on Chrome on Mac with uBlock Origin.

Sorry for not being more clear. The onboarding process is a simplified version of the actual scraper builder. The fields are not editable, it's just to get used to the scraping flow. I've noted down your suggestion and I'll make this more clear or add field edition. Thanks!

Yeah, this took me a minute to figure out, too. I'd change it to make it clear those are just static slides. Even better, remove it entirely, and when the user lands on the home page (after verifying), open a wizard that guides them through setting up their first scraper.

I also had a real problem naming the "Store as" field for my data extractor. It didn't seem to like things in the format "foo_bar_baz56" (i.e. ending with digits). This page https://mrscraper.freshdesk.com/support/solutions/articles/1... says "The variable name can not contain special signs" but doesn't explain what special signs are. Anything other than [A-Z_]?

Now that I've finally set up and tested my first scraper, I'm really impressed. It was much easier to set up than I would have guessed, and specifying a selector made it dead simple. Results worked out of the box, on a site that is super touch about being scraped.

However, now that I'm viewing my scraper, I see no way of editing the scraper or data extractor. What's the trick to editing a scraper once you've saved it and gone back to view it?

Re: Show HN: MrScraper – A visual web-scraping tool

#76

Post by SturgeonsLaw »

asdadsdad wrote: post #15
bdcravens wrote: post #2

This marketing bit seems a bit conflicting: "With MrScraper, you won't be blocked. We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio." "won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible…

also considering tests above - \"webDriver\": \"FAIL\" - seems like you'll totally get blocked by any anti-bot

My actual browser that I use as a human failed that test so it's probably more on them than anything.

Or I might have some kinda of addin/setting configured from hacking around on something over the years.

Re: Show HN: MrScraper – A visual web-scraping tool

#77

Post by buffer_overflow »

kmoser wrote: post #74
buffer_overflow wrote: post #51

Earlier quoted context omitted.

Sorry for not being more clear. The onboarding process is a simplified version of the actual scraper builder. The fields are not editable, it's just to get used to the scraping flow. I've noted down your suggestion and I'll make this more clear or add field edition. Thanks!

Yeah, this took me a minute to figure out, too. I'd change it to make it clear those are just static slides. Even better, remove it entirely, and when the user lands on the home page (after verifying), open a wizard that guides them through setting up their first scraper. I also had a real problem naming the "Store as" field for my data extractor. It didn't seem to like things in the format "foo_bar_baz56" (i.e. endi…

Thanks for your precious feedback!

I've noted everything down and I'll improve the onboarding experience, fix and make variable names more understandable and improve the edit button.

A scraper is not editable once is queued to run or currently running, then, you need to reload the page for the edit button to appear again. I will improve this.

Thank you again

Re: Show HN: MrScraper – A visual web-scraping tool

#78

Post by buffer_overflow »

jxramos wrote: post #62

Glad to see this text """ What happens if my scraping fails? Not to worry! We will make every effort to determine the cause of the problem and assist you in resolving any issues with your scraper. Additionally, please note that unsuccessful scrapings will not be included in your monthly quota. """ I'm curious the feedback mechanism for failed scrapes. Is there any validation configuration or an email notification I c…

If the target page or DOM changes, it's not considered a failed scraping, just the data you wanted will return empty.

But you gave me the idea to send an alert or email notification if a scraper stops returning data or content changes.

Thanks

Re: Show HN: MrScraper – A visual web-scraping tool

#79

Post by buffer_overflow »

marcopicentini wrote: post #65

On this page text field are disabled (Chrome, MacOsx) https://app.mrscraper.com/onboarding

The onboarding process is not a fillable form, it's just a simplification of the scraper builder to show how it works. You just have to click next.

Other users have also reported that this is a bit confusing, so I'm going to start working on improving this.

Thanks for you feedback!

Re: Show HN: MrScraper – A visual web-scraping tool

#80

Post by buffer_overflow »

rfeague wrote: post #54

I was hoping for something that would allow me to load a page to be scraped, mark the things I'm interested in, and have it help with the selection expressions.

I'm experimenting with something like you described, but there are lots of edge cases and needs a bit of polishing. But for now, I made a Chrome extension that helps tinkering with CSS selectors.
Post reply on HN