RoboBrowser: Your friendly neighborhood web scraper

#1

Post by pmoriarty » Thu, May 12, 2016, 1:43 AM UTC

RoboBrowser: Your friendly neighborhood web scraper
github.com

Re: RoboBrowser: Your friendly neighborhood web scraper

#2

Post by PhasmaFelis » Thu, May 12, 2016, 6:16 AM UTC

Could someone explain what this is for, maybe with a couple of examples? This is getting to be a problem on HN.

Re: RoboBrowser: Your friendly neighborhood web scraper

#3

Post by est » Thu, May 12, 2016, 6:22 AM UTC

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

Re: RoboBrowser: Your friendly neighborhood web scraper

#4

Post by foota » Thu, May 12, 2016, 6:23 AM UTC

est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

I know there's extensions that help already with generating finders for automated e2e testing, you might be able to use something like that.

Re: RoboBrowser: Your friendly neighborhood web scraper

#5

Post by tlrobinson » Thu, May 12, 2016, 6:27 AM UTC

PhasmaFelis wrote: post #2

Could someone explain what this is for, maybe with a couple of examples? This is getting to be a problem on HN.

Really? It's right there on the main Github page, a 3 sentence description and 6 code examples.

Re: RoboBrowser: Your friendly neighborhood web scraper

#6

Post by caseyf7 » Thu, May 12, 2016, 6:32 AM UTC

est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

The Resurrectio Chrome extension allows you to do this with PhantomJS

Re: RoboBrowser: Your friendly neighborhood web scraper

#7

Post by pkmishra » Thu, May 12, 2016, 6:50 AM UTC

What benefit does it provide in comparison to Scrapy?

Re: RoboBrowser: Your friendly neighborhood web scraper

#8

Post by pcr0 » Thu, May 12, 2016, 6:58 AM UTC

Hmm, I can see why I'd want to use this library over piecing together requests and BS4 myself for every project. I love how simple the examples look.

I have a project I'm working on that will involve scraping many different websites on a daily basis. My only scraping experience so far is using cheerio[0] to scrape a single page with a 1,000 row HTML table. Should I start with something BS-based like this or should I jump straight into Scrapy? Or are there any other alternatives I should try?

[0]: https://github.com/cheeriojs/cheerio

Re: RoboBrowser: Your friendly neighborhood web scraper

#9

Post by popey456963 » Thu, May 12, 2016, 7:21 AM UTC

est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver.

[0] http://www.seleniumhq.org/

Re: RoboBrowser: Your friendly neighborhood web scraper

#10

Post by tsergiu » Thu, May 12, 2016, 7:22 AM UTC

est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

Shameless plug, check out https://parsehub.com