Ask HN: Is there any HTML table scraper generator in python or else?

#1

Post by jeffjia » Mon, Oct 21, 2013, 1:37 AM UTC

Hi,

In one of my projects, I happen to need to get some scrapers running for tens of websites to collect rows, columns of tables (, , ). Those tables are well formatted. I have written several scrapers in python, which basically use CSS selector and then do some simple transformation with regular expression. I just wonder whether there is any scraper generator which may take a url and sample target output as input, and produce a scraper automatically?

Any suggestion is welcomed. Thanks in advance.

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#2

Post by murtza » Mon, Oct 21, 2013, 1:51 AM UTC

Have you taken a look at the Scrapy framework for Python?

http://scrapy.org/

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#3

Post by Johnie » Mon, Oct 21, 2013, 2:29 AM UTC

If you don't want to build it yourself, check out import.io. They turn any website into an API. They did a demo at SV Newtech a couple months ago.

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#4

Post by tonyfelice » Mon, Oct 21, 2013, 2:36 AM UTC

Have you looked at phantomjs?

The webintro example here (https://github.com/ariya/phantomjs/wiki/Examples) scrapes a specific element.

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#5

Post by taddeimania » Mon, Oct 21, 2013, 2:41 AM UTC

I've used BeautifulSoup to do stuff like this.

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#6

Post by Larrikin » Mon, Oct 21, 2013, 2:49 AM UTC

murtza wrote: post #2

Have you taken a look at the Scrapy framework for Python? http://scrapy.org/

I wrote a couple a few scrapers and found scrapy to be my best option

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#7

Post by jeffjia » Mon, Oct 21, 2013, 3:25 AM UTC

tonyfelice wrote: post #4

Have you looked at phantomjs? The webintro example here ( https://github.com/ariya/phantomjs/wiki/Examples ) scrapes a specific element.

I was using mechanizer + beautiful soup in python before, but it seems that this one also needs human to read the html source code to pick a css selector instead of automating it...

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#8

Post by jeffjia » Mon, Oct 21, 2013, 3:29 AM UTC

murtza wrote: post #2

Have you taken a look at the Scrapy framework for Python? http://scrapy.org/

Thanks. I used beautiful soup for the parser, and actually have written a crawler framework for my scenario. But I was wondering whether there is any tool that could automate the selection of css selector or xpath.

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#9

Post by jeffjia » Mon, Oct 21, 2013, 3:30 AM UTC

taddeimania wrote: post #5

I've used BeautifulSoup to do stuff like this.

Yeah. Me too. The css selector is quite convenient. The only problem is that I need to pick the selector set for each website I need to scrape, and there are tens of them, which makes the work itself time-consuming...

Re: Ask HN: Is there any HTML table scraper generator in python or else?

#10

Post by jeffjia » Mon, Oct 21, 2013, 3:44 AM UTC

Johnie wrote: post #3

If you don't want to build it yourself, check out import.io. They turn any website into an API. They did a demo at SV Newtech a couple months ago.

Thanks Johnie. It is almost what I want, except that it is not open-source and free...