11–20 of 61 posts
I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.
Does it support sites which require a JS enabled browser?
I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.
SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver. [0] http://www.seleniumhq.org/
Does it support sites which require a JS enabled browser?
Last time I needed something like this I used selenium. And I use requests the rest of the time.
Earlier quoted context omitted.
SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver. [0] http://www.seleniumhq.org/
One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).
You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others.
The only reason this is (in my opinion), better than other solutions is because you can see the physical webpage it's loading, and for the sheer ease of use that this has. You don't even really need any coding experience to get a simple test running.
What benefit does it provide in comparison to Scrapy?
Earlier quoted context omitted.
One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).
Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason t…
You can also use Selenium tests with things like https://www.browserstack.com/automate , where TL;DR they run your selenium test on dozens of browser + platform combinations and send you the results, like screenshots and any javascript errors. If you're familiar with CI stuff, you can see how powerful this has the potential to be. It's non-trivial but very possible to run your own cluster of selenium nodes as well; check out the official Selenium Grid: http://www.seleniumhq.org/projects/grid/
Could someone explain what this is for, maybe with a couple of examples? This is getting to be a problem on HN.
Really? It's right there on the main Github page, a 3 sentence description and 6 code examples.