Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

131–140 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#131

Post by voltagex_ »

ricardobeat wrote: post #126
randomdrake wrote: post #46

The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…

HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).

A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.

Re: Show HN: Kimono – Never write a web scraper again

#132

Post by chii »

pranade wrote: post #87
DanBlake wrote: post #83

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data?

of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

Re: Show HN: Kimono – Never write a web scraper again

#133

Post by pranade »

dikei wrote: post #130

I don't think this can beat the speed of a hand-tune crawler. When I write crawlers, I skip rendering page and javascript execution if it isn't needed, which massively speed up the crawling process.

There's definitely things that a custom-built scraper can do more efficiently than kimono, but our focus right now making scraping accessible across a broad enough range of web sites.

Re: Show HN: Kimono – Never write a web scraper again

#134

Post by pranade »

jlees wrote: post #114

I like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy task to make accessible and reliable -- good luck! It makes me wonder if there isn't a whole "API to web/mobile app with custom metadata" product in there somewhere. I can imagine a lot of folks starting to get into data analysis and pipelin…

Yes, we're excited about the possibilities of an end-to-end use case as well... in fact we were surprised when we found a lot of interest in front-end output layers on top of the APIs than just the APIs themselves. Would be curious to know what output features would be most valuable for you.

Re: Show HN: Kimono – Never write a web scraper again

#135

Post by pranade »

voltagex_ wrote: post #131
ricardobeat wrote: post #126

Earlier quoted context omitted.

HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).

A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.

Kimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.

Re: Show HN: Kimono – Never write a web scraper again

#136

Post by jval »

Great job guys.

One problem I've had though is that I think you guys are hosted on AWS - a lot of websites block incoming connections from AWS.

Are there plans to add an option in future to route through clean IPs? Premium or default, this would be cool and make it a lot more useful.

Re: Show HN: Kimono – Never write a web scraper again

#137

Post by eth »

Great tool!

I'm coming at things from a non-coder perspective and found it easy to use, and easy to export the data I collected into a usable format.

For my own enjoyment, I like to track and analyze Kickstarter project statistics. Options up until now have been either labor intensive (manually entering data into spreadsheets) or tech heavy (JSON queries, KickScraper, etc. pull too much data and my lack of coding expertise prevents me from paring it down/making it useful quickly and automagically) as Kickstarter lacks a public API. Sure, it is possible to access their internal API or I could use KickScraper, but did I mention the thing about how I dont, as many of you say, "code"?

What I do understand is auto-updating.CSV files, and that's what I can get from Kimono. Looking forward to continued testing/messing about with Kimono!

Re: Show HN: Kimono – Never write a web scraper again

#139

Post by rpedela »

Definitely awesome presentation and product.

The example doesn't seem to work right on Firefox. On Chrome, if I click "Character" in the table then it highlights the whole column and asks if I want to add the data in the column. On Firefox, clicking "Character" just highlights "Chatacter" and that is it.

Ubuntu 12.04

Firefox 25.0.1

Re: Show HN: Kimono – Never write a web scraper again

#140

Post by pranade »

rpedela wrote: post #139

Definitely awesome presentation and product. The example doesn't seem to work right on Firefox. On Chrome, if I click "Character" in the table then it highlights the whole column and asks if I want to add the data in the column. On Firefox, clicking "Character" just highlights "Chatacter" and that is it. Ubuntu 12.04 Firefox 25.0.1

Thanks for flagging. We'll get on this to figure out what's going on when running on ubuntu
Post reply on HN