Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RoboBrowser: Your friendly neighborhood web scraper

github.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: RoboBrowser: Your friendly neighborhood web scraper

#41

Post by thomasahle »

I'd like to write a small scraper for a website that uses NTLM authentication, the headers it sends are:

    HTTP/1.1 401 Unauthorized
    Server: Microsoft-IIS/8.5
    WWW-Authenticate: NTLM
    WWW-Authenticate: Negotiate
    ...
Does RoboBrowser support these kinds of protocols? I tried to get it to work with Scrapy, but it seemed non-trivial...

Re: RoboBrowser: Your friendly neighborhood web scraper

#43

Post by tekacs »

markbnj wrote: post #42

Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.

At a glance, no - it uses Requests to fetch pages and BeautifulSoup to parse them, the latter of which only parses the HTML into a document object.

So static HTML parsing only.

Re: RoboBrowser: Your friendly neighborhood web scraper

#44

Post by logn »

popey456963 wrote: post #16
enibundo wrote: post #14

Earlier quoted context omitted.

One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).

Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason t…

You can use my project https://github.com/machinepublishers/jbrowserdriver which can be both headless or (what's the opposite of headless?) have a full GUI. It runs on Java using Java's built-in browser and will take a second to warm up when an instance is created, but after doing so, you can reuse the existing browser since I added a reset() API. Performance is comparable to desktop browsers (slightly slower), and for every action it blocks until AJAX page loads finish.

Re: RoboBrowser: Your friendly neighborhood web scraper

#45

Post by yes_or_gnome »

markbnj wrote: post #42

Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.

I use PhantomJS as well, but (assuming you haven't already) look at CasperJS. It uses PhantomJS, but it is more friendly to use for bigger tasks.

Re: RoboBrowser: Your friendly neighborhood web scraper

#47

Post by nikolay »

brynedwards wrote: post #24
aexaey wrote: post #21

Earlier quoted context omitted.

It doesn't. To scrape (or fake-API) js-only websites you have to either: - drive a browser (firefox/chrome) via already mentioned here selenium/webdriver (potentially hiding the actual browser window into a virtual X by wrapping the whole thing with xvfb-run), - or use one of the webkit-based toolkits: phantomjs [1] or headless horseman [2]. There is also an interesting project that combines the two, i.e. it drives a…

I recently wrote a browser-driven scraper using Nightmare[1], which uses Electron under the hood. Another option for those who prefer python is dryscrape[2], although I haven't tried it. [1] https://github.com/segmentio/nightmare [2] http://dryscrape.readthedocs.io/en/latest/

Dryscrape is really cool! Thanks for sharing!

Re: RoboBrowser: Your friendly neighborhood web scraper

#48

Post by grewil »

Dolores12 wrote: post #33
aexaey wrote: post #25

I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman. [1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec... [2] https://pypi.python.org/pypi/mechanize/

Mechanize is outdated and python 2 only. We tried it and switched to RoboBrowser.

I find WWW::Mechanize::Firefox an interesting alternative. The API is similar to WWW::Mechanize, but it remote controls an instance of Firefox using the MozRepl addon. This means scraping/interacting with pages using JS works fine.

http://search.cpan.org/~corion/WWW-Mechanize-Firefox-0.78/li...

Re: RoboBrowser: Your friendly neighborhood web scraper

#49

Post by jimmaswell »

Dolores12 wrote: post #33
aexaey wrote: post #25

I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman. [1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec... [2] https://pypi.python.org/pypi/mechanize/

Mechanize is outdated and python 2 only. We tried it and switched to RoboBrowser.

In some significant ways, python 2 is the better language than 3, though.

Re: RoboBrowser: Your friendly neighborhood web scraper

#50

Post by dchuk »

Benfromparis wrote: post #40

Interesting for unprotected websites but it's easy to detect and to block: no valid js, no valid meta header, no valid cookie, suspect behavior... Selenium is a much "elaborated" solution, but still, can be detected most of the time. Disclosure: I'm DataDome co-founder. If you want to detect bad bots and scrapers on your website, don't hesitate to try out for free and to share your feedback with us https://datadome.c…

I realize you have reasons not to answer this question, but out of curiosity, what sorts of thing can tip off the fact that a site is getting scraped by a real browser and selenium?
Post reply on HN