Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Headless Web Browser Scraping on Amazon Linux

fruchterco.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Python Headless Web Browser Scraping on Amazon Linux

#21

Post by spikels »

swinglock wrote: post #18
fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

Aren't you stuck with JavaScript then? Sure, PhanthomJS is awesome, but Python is even in the title, so it's not just a side note.

Yes - Unless you are parsing static HTML you will need the rest of the browser's functionality which is implemented as a JavaScript engine. You will also need the original content from the website which will be in JavaScript.

In theory you could recreate this in another language such as Python but you would have to both parse the JavaScript from the website and implement a full browser.

Re: Python Headless Web Browser Scraping on Amazon Linux

#22

Post by ianhawes »

xfour wrote: post #12

One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.

Something to consider is that the trend the past year has been to use headless browsers over BeautifulSoup, cURL, etc.. because headless browsers are harder to detect by anti-scraping systems and can interpret JavaScript.

Re: Python Headless Web Browser Scraping on Amazon Linux

#23

Post by takluyver »

xfour wrote: post #12

One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.

BS4, which is still actively developed, got out of the parser game - it can now use lxml (fast) or html5lib (highly tolerant) to parse the HTML. It's kept the convenient interface to dig into the DOM, and it's kept the UnicodeDammit encoding detection system.

Re: Python Headless Web Browser Scraping on Amazon Linux

#24

Post by takluyver »

ianhawes wrote: post #22
xfour wrote: post #12

One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.

Something to consider is that the trend the past year has been to use headless browsers over BeautifulSoup, cURL, etc.. because headless browsers are harder to detect by anti-scraping systems and can interpret JavaScript.

That's what the OP is about ;-). But BeautifulSoup isn't a way to retrieve a web page, it's a way to parse HTML. You can get the page with a headless browser, and then transfer the DOM into a BeautifulSoup tree to do your scraping.

Re: Python Headless Web Browser Scraping on Amazon Linux

#25

Post by nirvdrum »

cstejerean wrote: post #20
nirvdrum wrote: post #13

Earlier quoted context omitted.

These aren't edge cases. They're asked about constantly. Most people are using Selenium because they care about everything on the page. Otherwise, your stdlib HTTP client would be sufficient. That aside, if PhantomJS already has the info, you can always fetch it with executeScript. If you do feel that strongly about the status code part though, I'd urge you to comment on the public draft of the W3C spec: http://www.w…

From the point of view of simulating actual users, the fact that some random third-party resource on the page failed to load is not particularly relevant. That happens all the time as I browse around the web, and I never have to care about it as long as the site continues to function. So it very much is an edge case compared to the page itself failing to load.

A JavaScript file failing to load will bork most pages. A CSS file failing to load or a key image will cause most people to quit. And an Ajax request failing in a single-page app will render it useless.

But, my point of view is from actual Selenium users. This is framed by providing support on the IRC channel, on the mailing lists, triaging the issue tracker, and by interacting with people at SeleniumConf and the local Boston meetup. It's not some fringe use case and I'm not arguing the point for the sake of arguing it. The original supposition that it's an edge case is not accurate. And sure, the web breaks. That's why people using Selenium would like a way to catch that. And that's a big part of why the BrowserMob Proxy project exists.

Re: Python Headless Web Browser Scraping on Amazon Linux

#26

Post by Wilya »

cinquemb wrote: post #15
kawsper wrote: post #5

Earlier quoted context omitted.

The file that he is fetching ( phantomjs-1.9.1-linux-x86_64.tar.bz2 ) is the executables for his platform, with some examples on usage and a readme.

That doesn't seem like a very safe thing to do... dont they have sc for PhantomJS one can checksum and run ./configure > make > sudo make install?

PhantomJS is pretty big. IIRC, building it takes quite some time. I think they bundle webkit and the necessary parts of Qt, and you'd have to be out of your mind to build that from source if you can avoid it.

Using official distribution packages would be a better idea, but their freshness can vary, especially on RHEL.

Re: Python Headless Web Browser Scraping on Amazon Linux

#27

Post by j-kidd »

Off topic: it is perfectly fine to install things like PyQt / PySide on a headless server. I suppose the problem is because the distro doesn't provide these packages?

Also, PhantomJS works fine in this case because the binary in the tarball is statically compiled. You can find a whole lot of qt stuffs inside PhantomJS source repository. There ain't no such thing as "truly headless".

Re: Python Headless Web Browser Scraping on Amazon Linux

#28

Post by techaddict009 »

Wow was searching something similar. Actually was trying to build a app which scraps data from movie ticket booking sites and provides data via SMS to user that whether tickets are still available or not. Because everyone doesn't have access to internet in India yet.

@Steven5158 thanks for the share.

If anyone here wants help in building SMS apps do contact me.

Re: Python Headless Web Browser Scraping on Amazon Linux

#29

Post by spikels »

nirvdrum wrote: post #9
fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

Well, I think the matter is a bit more complicated than that. When dealing with a full browser, you fetch a lot of resources. The status code for the first page fetch may be easily obtained, but your API gets very wonky as soon as you want to get status codes for all linked resources. Even if you managed that, any Ajax requests would complicate things, especially if they have deferred loading. And then you have WebSo…

Phantomjs handles everything you mention (status codes on large numbers of resources, ajax, deferred loading monitoring and HAR output) with the possible exception of websockets - I have not tried and very little documentation today but it should work. The big limitation is this is WebKit-only right now.

For example: here's the wiki on network monitoring including HAR: https://github.com/ariya/phantomjs/wiki/Network-Monitoring

The API seems pretty clean to me but I guess that is a matter of opinion.

Re: Python Headless Web Browser Scraping on Amazon Linux

#30

Post by keypusher »

We do quite a bit of web scraping / parsing on headless servers with Selenium. What we did was just install some X packages and run VNC server on the headless clients with Firefox. Cool thing about that is you can then go watch the scripts executing if you connect to the VNC session and take a screenshot on failure, etc.
Post reply on HN