Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

211–220 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#211

Post by ronaldx »

ricardobeat wrote: post #126
randomdrake wrote: post #46

The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…

HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).

This is awkward, but I disagree.

HN pages are very well structured and are probably an ideal case for an automatic scraper.

If your automatic scraper doesn't work on HN, it is unlikely to work generally.

Re: Show HN: Kimono – Never write a web scraper again

#212

Post by billmalarky »

pranade wrote: post #87
DanBlake wrote: post #83

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.

Re: Show HN: Kimono – Never write a web scraper again

#213

Post by pranade »

billmalarky wrote: post #212
pranade wrote: post #87

Earlier quoted context omitted.

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

I hope that the lite plan will feature auth handling, I can't imagine the service being useful in most cases without it.

We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free

Re: Show HN: Kimono – Never write a web scraper again

#214

Post by pranade »

littledot5566 wrote: post #177
bambax wrote: post #172

> Web scraping. It's something we all love to hate. You wish the data you needed to power your app, model or visualization was available via API. But, most of the time it's not. So, you decide to build a web scraper. You write a ton of code, employ a laundry list of libraries and techniques, all for something that's by definition unstable, has to be hosted somewhere, and needs to be maintained over time. I disagree.…

Perhaps this could be automated by finding the same content in two versions of the dom and then doing a diff on the structure, updating the rules?

It would be great to automate this eventually. For now, we're trying to make it really easy to set up and rebuild the scraper. If it goes down, you'll see it in the status on your user dashboard. We're also implementing alerts, so you can opt to get an email notification if a scrape fails

Re: Show HN: Kimono – Never write a web scraper again

#215

Post by walden42 »

pranade wrote: post #101
scotty79 wrote: post #70

Earlier quoted context omitted.

Another feature, simple one: Allow to add some filters to the data stream. For example: only posts that contain word "bitcoin" in the name or only those with 50 upvotes or more.

Thanks for the suggestion... adding to the list :)

Make sure to include regex matching =)

Re: Show HN: Kimono – Never write a web scraper again

#218

Post by pranade »

ricardobeat wrote: post #196
roel_v wrote: post #182

Earlier quoted context omitted.

OK so you're saying that instead of using a scraper to deal with malformed data out there (the whole reason of its existence), instead we should use a format that is better suited for machine representation? That's like saying 'yeah I've got this car here to take you to places that are very far away, too far to walk; except it doesn't work very well if you want to go far away, so you're betting off just staying at ho…

Scrapers exist to take data out of HTML, malformed or not, when an API or feed is not available. Yes, ideally HTML is written in a semantic, annotated machine-parseable way; that also enables smarter search engines, better accessibility, interoperability and so on. That's one of the main reasons behind the changes to the standard made in HTML5. A better example would be "hey I have this cool device that will improve…

Kimono can handle many pages with malformed/ old HTML. Of course, it's still beta and there are still pages that break it, so we're improving it as we go with the help of early adopters like everyone on this thread. Of course, our goal is an ideal state it works everywhere perfectly :)

Re: Show HN: Kimono – Never write a web scraper again

#220

Post by pranade »

walden42 wrote: post #215
pranade wrote: post #101

Earlier quoted context omitted.

Thanks for the suggestion... adding to the list :)

Make sure to include regex matching =)

Thanks. We support regex matching now. Try dragging to select text, if there's a relevant regex pattern and kimono will find it (there's an example inthe blog post). You can preview (and soon, you'll be able to edit) the CSS and Regex also in advanced mode.
Post reply on HN