I'm a RoR developer and have a project where I need to scrape a webpage, specifically a University's course list page, to populate a database. I've never scraped a webpage before, are there any pointers or guides that anyone can recommend? I don't even know where to start.
Ask HN: I want to scrape a web page...
1–10 of 18 posts
Re: Ask HN: I want to scrape a web page...
#2If you only need to scrape one University's one course list page to populate one database, your best bet is probably copy-paste with some tidying up with your text editor of choice. If you need to do more, I would look into Hpricot or Nokogiri, both Ruby libraries for parsing HTML. Then just download the page content, pass it to your parser of choice, and go wild.
Re: Ask HN: I want to scrape a web page...
#3If you only need to scrape one University's one course list page to populate one database, your best bet is probably copy-paste with some tidying up with your text editor of choice. If you need to do more, I would look into Hpricot or Nokogiri, both Ruby libraries for parsing HTML. Then just download the page content, pass it to your parser of choice, and go wild.
Second this. I wouldn't say I'm a RoR expert, but I've used both Hpricot and Nokogiri. I found Hpricot was a little easier to get going but Nokogiri ended up working out better for me. They work very in very similar fashions and are fairly straightforward.
Re: Ask HN: I want to scrape a web page...
#4Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library).
That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in tags, etc.
Let me know if you need more help. But I think as a first step write a program to download a web page.
Re: Ask HN: I want to scrape a web page...
#5If you only need to scrape one University's one course list page to populate one database, your best bet is probably copy-paste with some tidying up with your text editor of choice. If you need to do more, I would look into Hpricot or Nokogiri, both Ruby libraries for parsing HTML. Then just download the page content, pass it to your parser of choice, and go wild.
well... it's a big University... the largest in the country actually, so there are thousands and thousands of classes. I think copying/pasting may be time prohibitive.
Is it possible to setup a crawler that would iterate through what could be hundreds of pages, with the same HTML structure, and download/parse the page?
Re: Ask HN: I want to scrape a web page...
#6Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library). That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in tags, etc. Let me know if you need more help. But I think as a first step write a program t…
I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?
Re: Ask HN: I want to scrape a web page...
#7If you only need to scrape one University's one course list page to populate one database, your best bet is probably copy-paste with some tidying up with your text editor of choice. If you need to do more, I would look into Hpricot or Nokogiri, both Ruby libraries for parsing HTML. Then just download the page content, pass it to your parser of choice, and go wild.
well... it's a big University... the largest in the country actually, so there are thousands and thousands of classes. I think copying/pasting may be time prohibitive. Is it possible to setup a crawler that would iterate through what could be hundreds of pages, with the same HTML structure, and download/parse the page?
Yes. Look up Hpricot or Nokogiri and review the documentation. It will be clear how to get started. Basically you just give it a page and ask it what you want. If you want to handle multiple pages, just iterate through them like you'd iterate through anything else.
Re: Ask HN: I want to scrape a web page...
#8If you only need to scrape one University's one course list page to populate one database, your best bet is probably copy-paste with some tidying up with your text editor of choice. If you need to do more, I would look into Hpricot or Nokogiri, both Ruby libraries for parsing HTML. Then just download the page content, pass it to your parser of choice, and go wild.
well... it's a big University... the largest in the country actually, so there are thousands and thousands of classes. I think copying/pasting may be time prohibitive. Is it possible to setup a crawler that would iterate through what could be hundreds of pages, with the same HTML structure, and download/parse the page?
Yes, that is exactly what the crawler should do. I had no crawler experience a few months ago but wanted to do the same thing. I didn't know Ruby so I want the Java route. Now it's trivially easy, I just wrote a crawler last night to pull a bunch of info off hundreds of pages for my next app.
First start with how you yourself would get at each page of data. If it all starts from a head page figure out how to get to each link, follow those links and repeat until you are at the data pages. Then point your crawler to the head page and code in the patterns. If the pages are not all linked from a head page but have a url pattern figure out how to pull down the info needed to fill in the url pieces, pull that info down then visit each page filling in the pattern with the data you now have.
I can't give any Ruby guidance but if you have general questions on web scraping you can shoot me an email.
Re: Ask HN: I want to scrape a web page...
#9Earlier quoted context omitted.
well... it's a big University... the largest in the country actually, so there are thousands and thousands of classes. I think copying/pasting may be time prohibitive. Is it possible to setup a crawler that would iterate through what could be hundreds of pages, with the same HTML structure, and download/parse the page?
Yes, that is exactly what the crawler should do. I had no crawler experience a few months ago but wanted to do the same thing. I didn't know Ruby so I want the Java route. Now it's trivially easy, I just wrote a crawler last night to pull a bunch of info off hundreds of pages for my next app. First start with how you yourself would get at each page of data. If it all starts from a head page figure out how to get to e…
This is awesome... time to dive into hpricot!
Re: Ask HN: I want to scrape a web page...
#10Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library). That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in tags, etc. Let me know if you need more help. But I think as a first step write a program t…
I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?
require 'net/http'
require 'uri'
output = Net::HTTP.get(URI.parse("http://www.google.com"))