Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

151–160 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#151

Post by pranade »

thinkzig wrote: post #148

Great work so far. The tool was very intuitive and easy to use. My suggestion: once I've defined an API, let me apply it to multiple targets that I supply to you programatically. The use case driving my suggestion: I'm an affiliate for a given eCommerce site. As an affiliate, I get a data feed of items available for sale on the site, but the feed only contains a limited amount of information. I'd like to make the dat…

Thanks a ton for the feedback. Getting data from multiple similarly structured URLs programmatically is something we're working on now. We love hearing about the use cases you want to use this for so we can make sure we build out the right features to make kimono useful for you.

Re: Show HN: Kimono – Never write a web scraper again

#152

Post by pranade »

joshmlewis wrote: post #150

Is there an ability to scrape more than one page of data?

Not yet, but it's our #1 feature request, so we're working on it now. For now, you can make multiple APIs (one for each URL). If the URL takes query parameters though, you can re-use the same API and programmatically cycle through query parameters.

Re: Show HN: Kimono – Never write a web scraper again

#153

Post by GigabyteCoin »

I'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to take off?

I could have really used a service like this just yesterday actually, I ended up fiddling around with iMacros and got about 80% of what I was trying to achieve.

Re: Show HN: Kimono – Never write a web scraper again

#154

Post by pranade »

GigabyteCoin wrote: post #153

I'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to take off? I could have really used a service like this just yesterday actually, I ended up fiddling around with iMacros and got about 80% of what I was trying to achieve.

It's a great question. What we're really trying to do is make data accessible programmatically and at scale. We want to connect data providers and data consumers with APIs in a way that's mutually beneficial vs. being a tool for data theft. Our hope is to (once we scale) actually work with data providers directly on the on the distribution of their data so the IP ban becomes a non-issue.

Re: Show HN: Kimono – Never write a web scraper again

#155

Post by danso »

losvedir wrote: post #94
dunham wrote: post #67

The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages. The software is abandoned, but their algorithms are described in a paper: http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf

Oh, hey, memories. I worked one summer with David Huynh (who you're linking to there) and David Karger (his thesis advisor) on one of the Simile projects. I vaguely remember playing around with this tool you mentioned. I thiiiiink it was this one[0], although it seems to be superseded by this one[1] now. [0] http://simile.mit.edu/wiki/Piggy_Bank [1] http://simile.mit.edu/wiki/Sifter

Just had to chime in and say that David Huynh and his fellow programmers will be forever heroes to me and a small group of data journalists who depended on Gridworks/Google Refine/OpenRefine

Re: Show HN: Kimono – Never write a web scraper again

#156

Post by voltagex_ »

pranade wrote: post #135
voltagex_ wrote: post #131

Earlier quoted context omitted.

A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.

Kimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.

Sounds good. I've made a note of it for the next time I need to screen scrape something.

Re: Show HN: Kimono – Never write a web scraper again

#157

Post by wilg »

ricardobeat wrote: post #126
randomdrake wrote: post #46

The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…

HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).

I really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup?

I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible.

I can't tell you how often I click next page to find that I've taken too long and my session or whatever has expired.

Re: Show HN: Kimono – Never write a web scraper again

#158

Post by deskglass »

fizx wrote: post #125
tectonic wrote: post #116

Just use http://selectorgadget.com

You should write a blog post on lessons learned when we spent a year making ~this in 2008.

Thanks so much for creating SelectorGadget! I used it a lot when scraping some fanfiction and Wikipedia data.

Re: Show HN: Kimono – Never write a web scraper again

#159

Post by garyjob »

the_french wrote: post #149
chii wrote: post #132

Earlier quoted context omitted.

how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data? of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?

The route of proxying the web page presents much difficulty in doing actual authentication on Facebook or Google's website via the proxied webpage without first rewriting most of the javascript and hijacking their Ajax calls on the fly.

The approach I took was to hijack the Cookies from the browser once the user has signed in after on e.g. Facebook via the browser extension.

The route of proxying the website does in fact do away with the need to install any external 3rd libraries.

This browser extension I built coupled with the web service its integrated to does allow for scraping of pages from Facebook, Google and LinkedIn logged in pages as well.

https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...

Re: Show HN: Kimono – Never write a web scraper again

#160

Post by garyjob »

thatthatis wrote: post #120
pranade wrote: post #60

Earlier quoted context omitted.

Great suggestion... thanks for this one. We're putting this on our list

Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.

You could try doing IP address rotation by rotation EC2 instances or some other cloud services.

I wrote a library for that.

https://github.com/KrakeIO/resque-my-aws

Post reply on HN