Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What info do you web scrape for?

news.ycombinator.com

Post reply on HN
Print view View original

111–115 of 115 posts

Re: Ask HN: What info do you web scrape for?

#111

Post by carucez »

jasallen wrote: post #68

I am currently scraping for brand product and nutrition data. Having to build custom scrapers per brand is hell. I have a dream to use something closer to OCR against a rendered page, rather than parsing DOM. That way it would be less custom, and I could say, for instance, "find 'protein', the thing to the right of that is the protein grams". I, personally, don't know how to do this, but I'd be willing to pay for a m…

take a block of text, sentence, or paragraph.

build 2 classifiers: #1 that classifies a sentence that contains the information you want, and #2 that classifies the actual data within that subset.

NLP

Re: Ask HN: What info do you web scrape for?

#112

Post by hallz »

I made a javascript based automated scraper in a win7/vista desktop gadget. It was originally for dislplaying the remaining credit on my mobile. You put in search terms and a website and it scrapes it and tries to return what it thinks you are looking for (weather, stock prices, remaining blance etc). It works ok. I think there is definetly demand for a well made scraper/alert app/service though.

App is here http://robotification.com/creditile/

Also didn't yahoo make a thing for scraping this 'Yahoo Pipes' or something.

Re: Ask HN: What info do you web scrape for?

#113

Post by davidy123 »

hatethis wrote: post #101

Frankly I'm annoyed to see this topic here. Most people who have taken to scraping are low-life scum. They see content that others have spent months or years producing, simply set up a site that aggregates all of that information, and then sit back and collect revenue from ads or reseller links you paste everywhere. People who put in a few hours of work to take advantage of other people's hard work piss me off. :/

Are you against hyperlinks too? Seriously, computers naturally make content reusable. Try to imagine the next level when we don't depend on hoarding. The semantic web imagined this, though it was too complicated. But the idea still has huge benefits; every web site a linked database, with content precisely described. But I think many orgs are too afraid they don't really have something to offer in the big picture (that's what so much of business is about).

I do a lot of work in scraping, but it's for non profit healthcare, academic, and general knowledge augmentation. It's painful, but the only way to get to the next level without waiting a thousand years for everyone to make their own consistent API and metadata descriptions.

Re: Ask HN: What info do you web scrape for?

#114

Post by imns »

jasallen wrote: post #68

I am currently scraping for brand product and nutrition data. Having to build custom scrapers per brand is hell. I have a dream to use something closer to OCR against a rendered page, rather than parsing DOM. That way it would be less custom, and I could say, for instance, "find 'protein', the thing to the right of that is the protein grams". I, personally, don't know how to do this, but I'd be willing to pay for a m…

There's companies out there that already have this info. Checkout http://kwikeesystems.com .
Post reply on HN