Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RoboBrowser: Your friendly neighborhood web scraper

github.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: RoboBrowser: Your friendly neighborhood web scraper

#21

Post by aexaey »

r1k wrote: post #12

Does it support sites which require a JS enabled browser?

It doesn't. To scrape (or fake-API) js-only websites you have to either:

- drive a browser (firefox/chrome) via already mentioned here selenium/webdriver (potentially hiding the actual browser window into a virtual X by wrapping the whole thing with xvfb-run),

- or use one of the webkit-based toolkits: phantomjs [1] or headless horseman [2].

There is also an interesting project that combines the two, i.e. it drives a Firefox (or, more precisely, slightly outdated version of Gecko) to emulate a phantomjs-compatible API. [3]

phantomjs/slimerjs are pretty popular and even have tools that run on top of them, such as casperjs [4], that geared more to automated website testing, but can be quite good at scraping or fake-APIing too.

[1] http://phantomjs.org/

[2] https://github.com/johntitus/node-horseman

[3] https://slimerjs.org/

[4] http://casperjs.org/

Re: RoboBrowser: Your friendly neighborhood web scraper

#23

Post by mathheaven »

PhasmaFelis wrote: post #19
tlrobinson wrote: post #5

Earlier quoted context omitted.

Really? It's right there on the main Github page, a 3 sentence description and 6 code examples.

I know, I read it. It's for "browsing the web without a standalone web browser," and I'm sure that if that was something I had needed, I would have said "Oh! How lovely!" But, since I didn't have that need already, I'm not clear why someone would want that. And I'd like to know! So could you give me a couple of practical use cases? "User stories," if you're into that?

automation: go to this page, fill in the form, push submit, receive the result, process the result, send the processed result to another program for further analysis, finally emit and alert when attention is needed.

Re: RoboBrowser: Your friendly neighborhood web scraper

#24

Post by brynedwards »

aexaey wrote: post #21
r1k wrote: post #12

Does it support sites which require a JS enabled browser?

It doesn't. To scrape (or fake-API) js-only websites you have to either: - drive a browser (firefox/chrome) via already mentioned here selenium/webdriver (potentially hiding the actual browser window into a virtual X by wrapping the whole thing with xvfb-run), - or use one of the webkit-based toolkits: phantomjs [1] or headless horseman [2]. There is also an interesting project that combines the two, i.e. it drives a…

I recently wrote a browser-driven scraper using Nightmare[1], which uses Electron under the hood. Another option for those who prefer python is dryscrape[2], although I haven't tried it.

[1] https://github.com/segmentio/nightmare

[2] http://dryscrape.readthedocs.io/en/latest/

Re: RoboBrowser: Your friendly neighborhood web scraper

#25

Post by aexaey »

I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman.

[1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec...

[2] https://pypi.python.org/pypi/mechanize/

Re: RoboBrowser: Your friendly neighborhood web scraper

#27

Post by aavotins »

PhasmaFelis wrote: post #19
tlrobinson wrote: post #5

Earlier quoted context omitted.

Really? It's right there on the main Github page, a 3 sentence description and 6 code examples.

I know, I read it. It's for "browsing the web without a standalone web browser," and I'm sure that if that was something I had needed, I would have said "Oh! How lovely!" But, since I didn't have that need already, I'm not clear why someone would want that. And I'd like to know! So could you give me a couple of practical use cases? "User stories," if you're into that?

A dead simple example - scrape data from a webpage that doesn't have an API. You could down the wrong route of trying to parse the HTML and end up implementing a lot of logic manually OR you could use this wonderful library.

Re: RoboBrowser: Your friendly neighborhood web scraper

#28

Post by enibundo »

seanp2k2 wrote: post #18
popey456963 wrote: post #16

Earlier quoted context omitted.

Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason t…

You can record tests in Selenium or something like Capybara and replay them using something like PhantomJS, which is a headless browser-like JS execution environment that does things like generate a would-be DOM: https://github.com/jnicklas/capybara You can also use Selenium tests with things like https://www.browserstack.com/automate , where TL;DR they run your selenium test on dozens of browser + platform combinati…

what is CI? ...CLI?

Re: RoboBrowser: Your friendly neighborhood web scraper

#30

Post by shrikant »

enibundo wrote: post #28
seanp2k2 wrote: post #18

Earlier quoted context omitted.

You can record tests in Selenium or something like Capybara and replay them using something like PhantomJS, which is a headless browser-like JS execution environment that does things like generate a would-be DOM: https://github.com/jnicklas/capybara You can also use Selenium tests with things like https://www.browserstack.com/automate , where TL;DR they run your selenium test on dozens of browser + platform combinati…

what is CI? ...CLI?

Continuous Integration.
Post reply on HN