51–60 of 61 posts
Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.
I've done a lot of work scraping various sites and I can tell you this: basing any product on your ability to aggregate data via scraping will not work in the long run.
Eventually you will be asked not to scrape and then you'll get sued if you don't stop.
Case law is not in your favor here. See Craigslist Vs. 3Taps.
Earlier quoted context omitted.
In some significant ways, python 2 is the better language than 3, though.
In what ways?
1. More modules for Python2 than Python3. A lot of projects are forced to go down to Python2 to allow them to use the modules they want to use. 2. It's what they're familiar with. Generally the older developers like using Python2 because they know all what they're getting, no matter what strings are attached. 3. Better syntax. Apparently some people enjoy the Python2 syntax more than the Python3 one. Not using brackets for print statements seem to be the biggest plus, even though in my opinion it looks more Pythonic.
I'd like to write a small scraper for a website that uses NTLM authentication, the headers it sends are: HTTP/1.1 401 Unauthorized Server: Microsoft-IIS/8.5 WWW-Authenticate: NTLM WWW-Authenticate: Negotiate ... Does RoboBrowser support these kinds of protocols? I tried to get it to work with Scrapy, but it seemed non-trivial...
Earlier quoted context omitted.
Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason t…
You can use my project https://github.com/machinepublishers/jbrowserdriver which can be both headless or (what's the opposite of headless?) have a full GUI. It runs on Java using Java's built-in browser and will take a second to warm up when an instance is created, but after doing so, you can reuse the existing browser since I added a reset() API. Performance is comparable to desktop browsers (slightly slower), and f…
- Headless vs. Not - JS support vs. Not
I put a repo together with a sample script [2] for scraping leads off of a website which I will not name, but whose name rhymes with 'help'. It uses the PhantomJS browser for headless JS support. It also includes a Vagrantfile so you can avoid installing all the dependencies on your local machine.
Earlier quoted context omitted.
SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver. [0] http://www.seleniumhq.org/
One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).
I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.
Several startups have tried this. http://kimonolabs.com was is one I had experimented with and it was recently bought and shut down by Palantir.
I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman. [1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec... [2] https://pypi.python.org/pypi/mechanize/