Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

221–230 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#221

Post by Houshalter »

precisioncoder wrote: post #192
wilg wrote: post #157

Earlier quoted context omitted.

I really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup? I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible. I can't tell you how often I click next page to find that I've taken too long…

I like HN because it gives me exactly what I want. A list of interesting links, ranked by what people that I truest more than news sites think of them. Oh and offers comment sections for each of them. HN is wonderful link discovery during compile time or if you need a break.

A minimalist interface is fine but parts of HN are simply broken or suboptimal.

Re: Show HN: Kimono – Never write a web scraper again

#222

Post by automately »

DanBlake wrote: post #83

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

Creator of Automately here, our service could definitely be something you might be interested in. While we aren't directly in the business of web scraping, we do have a powerful automation service that can accomplish those needs using simple javascript and our powerful scalable automation API.

We are accepting early access requests right now. Check us out! http://automate.ly/

Re: Show HN: Kimono – Never write a web scraper again

#223

Post by LordIllidan »

romaniv wrote: post #203
LordIllidan wrote: post #202

Earlier quoted context omitted.

I think we all like HN, otherwise we wouldn't be here. That doesn't mean it can't improve.

If "improve" would involve turning it into AJAX-heavy app with images, CSS effects and some weird-ass scrolling interceptors, that requires me to load 200+ files for seeing the main page, I would rather live without any improvements. I think better terms might be "clean up" or "restructure".

And why do you assume that is what I meant by "improve"?

Re: Show HN: Kimono – Never write a web scraper again

#226

Post by pranade »

loceng wrote: post #166
pranade wrote: post #144

Earlier quoted context omitted.

At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt

I would say how you're scraping differs from say how Google, a search engine, scrapes. I'm not sure there is a way in robots.txt to define for each use? Knowing the data in a structured way, but then allowing it to be displayed in full off-site is quite different than using the scraped data for linking into a website.

Feedback from webmasters is really helpful for us. We want to make sure we're making data available via API responsibly, so would love to hear your suggestions/ thoughts as we define a scalable solution.

Re: Show HN: Kimono – Never write a web scraper again

#227

Post by pranade »

jlees wrote: post #167
pranade wrote: post #134

Earlier quoted context omitted.

Yes, we're excited about the possibilities of an end-to-end use case as well... in fact we were surprised when we found a lot of interest in front-end output layers on top of the APIs than just the APIs themselves. Would be curious to know what output features would be most valuable for you.

Well. Think about spreadsheets. Think about live spreadsheets powered directly by APIs. Bingo! I'm doing a couple of data mining projects right now and simply being able to query and look into the API outputs, as well as my local database, without building a custom frontend would've saved me a bunch of time. But I'm thinking more of the knowledge worker, or even the power user who wants to view their Fitbit, Up and L…

Love your idea. Would love to follow up on this with you

Re: Show HN: Kimono – Never write a web scraper again

#228

Post by SkyMarshal »

theg2 wrote: post #199
tinco wrote: post #187

Earlier quoted context omitted.

No. This site isn't about UX or innovation, it's about tech start ups. It's a constant reminder that something can be successful even if it was written in a LISP dialect and has a bunch of UX misses as long as the core idea is valid and the product usable enough. I've used a bunch of HN skins that were supposedly better designed, but none of them stuck. Apparently it's just plain unnecessary for HN to be better.

So because it's written in LISP it gets a free pass in other areas? That seems like a dubious claim.

No, despite it being written in an obscure language and sparse UI it has massive traction anyway. There's a lesson in there somewhere.

Re: Show HN: Kimono – Never write a web scraper again

#229

Post by scotty79 »

pranade wrote: post #220
walden42 wrote: post #215

Earlier quoted context omitted.

Make sure to include regex matching =)

Thanks. We support regex matching now. Try dragging to select text, if there's a relevant regex pattern and kimono will find it (there's an example inthe blog post). You can preview (and soon, you'll be able to edit) the CSS and Regex also in advanced mode.

Too bad I couldn't edit selectors and regex-es at this step. I could implement the filters I needed myself manually like this.

Re: Show HN: Kimono – Never write a web scraper again

#230

Post by Nilzor »

pranade wrote: post #135
voltagex_ wrote: post #131

Earlier quoted context omitted.

A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.

Kimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.

Have you thought of implementing a system for custom handling of edge cases? Realistically you can't handle all edge cases, so leaving the last 2% up to the user (which is a developer) would be a good idea. It'd still save him 98% of the work, but also give him the comfort of knowing it won't break even if site X is doing something weird with their HTML in the future.

I'll leave it to you to figure out how one would implement such a feature ;-)

Post reply on HN