Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

91–100 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#91

Post by tlianza »

dunham wrote: post #67

The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages. The software is abandoned, but their algorithms are described in a paper: http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf

If you're interested in hosted solutions that try to do automatic identification of pages, diffbot is worth a look. We've had some good experiences: http://diffbot.com/

Re: Show HN: Kimono – Never write a web scraper again

#92

Post by pranade »

pknight wrote: post #85

That UI made me go wow, this could be an awesome tool. Idea that pops into my mind is being able to grab data from those basic local sites run by councils, local news papers etc and putting it into a useful app. How dedicated are you guys to making this work because I'd imagine there are quite a few technical hurdles in keeping a service like this working long term while not getting blocked by various sites?

Love your suggestion. We're committed to making kimono better and we're working on it all the time. We want to make sure it's a responsible scraper, so want to work together with webmasters in cases where there might be blocking but the data is legal to share...

Re: Show HN: Kimono – Never write a web scraper again

#93

Post by shekyboy »

pranade wrote: post #86
shekyboy wrote: post #82

Like the parameter passthrough feature. Take a look at places where the parameters are part of the URL structure. For example a Target product page http://www.target.com/p/men-s-c9-by-champion-impact-athletic... In order to get data for a different product, I will have to modify the URL itself. I think same holds true for blog posts.

Yes, it's a great point. We're working on updating the query param passthrough to handle params within the URL structure.

Apart from that here are other items. You may have these on your list, but can count my vote to prioritize.

1. Pagination 2. Image URLs 3. Focus on page types such as product pages, posts etc. That way its easy to go from content to content. Will help crawling too 4. Link back to original page included in JSON

Finally common sites/pages used by multiple users of your systems should not count against the API count requirement under pricing. You may want to charge against total calls, like Parse.

Re: Show HN: Kimono – Never write a web scraper again

#94

Post by losvedir »

dunham wrote: post #67

The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages. The software is abandoned, but their algorithms are described in a paper: http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf

Oh, hey, memories. I worked one summer with David Huynh (who you're linking to there) and David Karger (his thesis advisor) on one of the Simile projects.

I vaguely remember playing around with this tool you mentioned. I thiiiiink it was this one[0], although it seems to be superseded by this one[1] now.

[0] http://simile.mit.edu/wiki/Piggy_Bank [1] http://simile.mit.edu/wiki/Sifter

Re: Show HN: Kimono – Never write a web scraper again

#96

Post by handelaar »

Maxious wrote: post #66
rlpb wrote: post #59

Are you familiar with ScraperWiki? I'm wondering how your work fits in with it. Edit: looks like they've moved away from that space, but have an old version available at: https://classic.scraperwiki.com/

The people who scrape data to avoid paying for APIs are the same people who will not pay for a service to make scraping easier ;)

The people who scrape data at Scraperwiki -- which was made by the same people who opened up parliamentary transcripts in the UK for the first time, and the UN's proceedings, and data about how MPs in London vote -- generally don't have an option to buy anything because the data's hidden by governments from the people who paid for it, on purpose.

But by all means take this opportunity to dismiss all of us as freeloaders.

Re: Show HN: Kimono – Never write a web scraper again

#98

Post by pranade »

fsckin wrote: post #84
pranade wrote: post #76

Earlier quoted context omitted.

Thanks for letting us know. Just tried and am getting the same error. The page is loading content dynamically from another source... We'll look into this and see if we can get it working on this page

Do you support POSTs for fetching dynamic data? I found where it's pulling from, here's the curl command: curl " https://www.keepandshare.com/calendar/fns_asynch_api.php?r=0... --data "action=getrange&i=1940971&from=2013-12-26&to=2014-02-06"

No, we don't have POST support quite yet. We're working on a solution.

Re: Show HN: Kimono – Never write a web scraper again

#99

Post by pranade »

ewebbuddy wrote: post #97

Really cool idea and tool. Still need to test this out properly. Is it possible to scrape note just one page but a stack of them? For example - a product catalog of 1000 SKUs extending upto 50pages.

We don't support that quite yet. It's our #1 feature request though, and we're working to get it ready soon

Re: Show HN: Kimono – Never write a web scraper again

#100

Post by AznHisoka »

Maxious wrote: post #66
rlpb wrote: post #59

Are you familiar with ScraperWiki? I'm wondering how your work fits in with it. Edit: looks like they've moved away from that space, but have an old version available at: https://classic.scraperwiki.com/

The people who scrape data to avoid paying for APIs are the same people who will not pay for a service to make scraping easier ;)

Actually, I usually scrape data because there is NO API I can pay for.
Post reply on HN