Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crawl a website with scrapy and store extracted results with MongoDB

isbullsh.it

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Crawl a website with scrapy and store extracted results with MongoDB

#12

Post by JackC »

For really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination:

  import httplib2, lxml, pyquery
  h = httplib2.Http(".cache")
  def get(url):
      resp, content = h.request(  url, headers={'cache-control':'max-age=3600'})
      return pyquery.PyQuery( lxml.etree.HTML(content) )
This gives you a little function that fetches any URL as a jquery-like object:

  pq = get("http://foo.com/bar")
  checkboxes = pq('form input[type=checkbox]')
  nextpage = pq('a.next').attr('href')
And of course all of the requests are cached using whatever cache headers you want, so repeated requests will load instantly as you iterate.

Just something else to throw in the toolbelt ...

Re: Crawl a website with scrapy and store extracted results with MongoDB

#13

Post by jat1 »

BaltoRouberol wrote: post #10
jat1 wrote: post #9

Also check this out for a pretty good discussion on scraping http://pyvideo.org/video/609/web-scraping-reliably-and-effic...

Yeah, I actually learnt scraping from Asheesh :) He's awesome.

I have been playing with scraping for quite some time now and have my own scripts and stuff, but I found that video informing and there were a few useful snippets I had missed.

Keep meaning to check out more of the Pycon vids

Re: Crawl a website with scrapy and store extracted results with MongoDB

#15

Post by the_cat_kittles »

JackC wrote: post #12

For really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination: import httplib2, lxml, pyquery h = httplib2.Http(".cache") def get(url): resp, content = h.request( url, headers={'cache-control':'max-age=3600'}) return pyquery.PyQuery( lxml.etree.HTML(content) ) This gives you a little function that fetches any URL as a jquery-like object: pq = get("http://foo.com/bar") checkboxes = pq('form inp…

Have checked out kenneth reitz's requests? Its fantastic, you might like it

Re: Crawl a website with scrapy and store extracted results with MongoDB

#16

Post by codehenge »

the_cat_kittles wrote: post #15
JackC wrote: post #12

For really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination: import httplib2, lxml, pyquery h = httplib2.Http(".cache") def get(url): resp, content = h.request( url, headers={'cache-control':'max-age=3600'}) return pyquery.PyQuery( lxml.etree.HTML(content) ) This gives you a little function that fetches any URL as a jquery-like object: pq = get("http://foo.com/bar") checkboxes = pq('form inp…

Have checked out kenneth reitz's requests? Its fantastic, you might like it

Link, for the interested:

https://github.com/kennethreitz/requests

Re: Crawl a website with scrapy and store extracted results with MongoDB

#17

Post by the_cat_kittles »

codehenge wrote: post #16
the_cat_kittles wrote: post #15

Earlier quoted context omitted.

Have checked out kenneth reitz's requests? Its fantastic, you might like it

Link, for the interested: https://github.com/kennethreitz/requests

thanks, i should have included a code sample too:

    import requests
    from lxml import etree

    jquery_like_page = etree.HTML(requests.get('url').text)
Post reply on HN