Ask YC: Grabbing content

#1

Post by qwestion » Thu, Apr 17, 2008, 5:25 AM UTC

First of all I would like to thank some of you who offered some life saving advice for my previous question.

Now that we have pulled up our socks and are making some headway, we have run into our first content problem.

We are building a business listings directory for a niche market. We have identified a bunch of ad-hoc sources for sourcing content. We have manually gathered content for our mockup, but to take it to the next level we need a lot more content than what we have going.

What are the different techniques used to gather content? It would be great if some of you have had specific experience in gathering content for some kind of a business listings directory.

Looking forward to some valuable insight!

Thanks again!

Re: Ask YC: Grabbing content

#2

Post by thorax » Thu, Apr 17, 2008, 5:41 AM UTC

Hmm, I really don't know how to help. I did just inquire into something not specifically related but might be randomly useful.

There's a company called DeepData @ http://www.deepdata.com which might be of some use? They have APIs for business listing queries, and may be able to help on some categorizations.

Re: Ask YC: Grabbing content

#3

Post by lux » Thu, Apr 17, 2008, 5:42 AM UTC

If you mean screen-scraping, isn't this roughly what you're looking for?

http://www.crummy.com/software/BeautifulSoup/

Re: Ask YC: Grabbing content

#4

Post by kleevr » Thu, Apr 17, 2008, 5:53 AM UTC

lixto

Re: Ask YC: Grabbing content

#5

Post by FiReaNG3L » Thu, Apr 17, 2008, 6:15 AM UTC

Dapper.net is your friend

Re: Ask YC: Grabbing content

#6

Post by davidw » Thu, Apr 17, 2008, 7:11 AM UTC

Just be very very careful if you grab any application forms;-)

Re: Ask YC: Grabbing content

#7

Post by spage » Thu, Apr 17, 2008, 12:54 PM UTC

My favorite technique is:

wget URL > HTML tidy HTML > XHTML xslt [identity transform based content extraction] XHTML > XML XML > DB

The whole process glued together with PERL or shell scripts. Depending on how you construct your content extraction, this technique can weather lots of the inevitable content style changes and easily adjusts when changes need to be made.