Wow looks amazing. I tried doing some queries on public directories, and it even supports parameter passing. Will be using this for some side projects.
Show HN: Kimono – Never write a web scraper again
51–60 of 234 posts
Re: Show HN: Kimono – Never write a web scraper again
#52I've written more web scraping code than I care to admit. A lot of the apps that ran on chumby devices used scraping to get their data (usually(!) with the consent of the website being scraped) since the device wasn't capable of rendering html (it eventually did get a port of Qt/WebKit, but that was right before it died and it wasn't well integrated with the rest of the chumby app ecosystem). This service looks great…
Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system that consolidates scrapes across users and hits sites (and pages) at human-like intervals. the second is to create a portal for webmasters that allows them special privileged access to analytics on data being extracted from their sites, and the ability to "turn…
Use a user-agent containing a URL to find out who and what you are, and honor my robots.txt.
Having a panel for webmasters along with that would be fine.
Re: Show HN: Kimono – Never write a web scraper again
#53Cool concept. One concern I'd have about this type of tool is that when it encounters something it can't handle, I'm stuck. Writing your own scraper means that you can modify it when you need to. I think the ultimate solution would be something like Kimono with the ability to write snippets of custom javascript to pull out anything that it can't handle by default.
Re: Show HN: Kimono – Never write a web scraper again
#54"Never write a web scraper again"... yea right.. sick and tired of such gimmicks and self promotion on the net today.
Re: Show HN: Kimono – Never write a web scraper again
#55Earlier quoted context omitted.
Thanks... yes, public data from governments is a great use case. Often a lot of apps built using scrapers will wind up driving up traffic/ sales a the source site so it's okay. We want to do responsible web scraping, so will respect webmasters robots.txt files to make sure it's legal.
I love the execution, but I also see inherent problems. Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service. You will encounter terms along the lines of: "Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing: U…
You've got a valid point. We want to eventually create a space that allows responsible scraping - so webmasters can have access to analytics on what's being scraped and can explicitly turn off kimono APIs for their domains if they see fit. We also think there are use cases for people who own their own data. Often, APIs will provide a way for companies to streamline their internal app development and figure out what to expose to the developer community before investing in an expensive API deployment.
Re: Show HN: Kimono – Never write a web scraper again
#56Cool concept. One concern I'd have about this type of tool is that when it encounters something it can't handle, I'm stuck. Writing your own scraper means that you can modify it when you need to. I think the ultimate solution would be something like Kimono with the ability to write snippets of custom javascript to pull out anything that it can't handle by default.
We're in the middle of implementing a more power developer version of the tool to handle the use cases you're talking about. The beginning of this is surfaced under the "advanced" tab in the data model view where we show the selectors and regular expressions that are produced. we want to ultimately let you edit those to customize the extractor. From there it'd be super cool to implement the javascript snippet feature you suggested.
Re: Show HN: Kimono – Never write a web scraper again
#57I built something very similar last year, but sadly never got around to polishing and launching it: http://exfiltrate.org/
(There's a prototype of an API generator hidden in a menu somewhere but it's nowhere near production ready)
Re: Show HN: Kimono – Never write a web scraper again
#58The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
Sorry you had a bad first experience. We've tested this on a lot of sites, and it works stably across a lot of different cases, but haven't solved it everywhere. Thanks for letting us know about the discussion page. We'll look into the bugs right now
Having written a whole lot of crawling code throughout the years, I can totally understand how monumental of a task this is. This really does look like a cool product. Glad you're actively hunting down ways to improve the demo before asking for money :)
Feel free to drop me a line if you want any specifics about the troubles I had.
Re: Show HN: Kimono – Never write a web scraper again
#59Are you familiar with ScraperWiki? I'm wondering how your work fits in with it.
Edit: looks like they've moved away from that space, but have an old version available at: https://classic.scraperwiki.com/
Re: Show HN: Kimono – Never write a web scraper again
#60Earlier quoted context omitted.
Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system that consolidates scrapes across users and hits sites (and pages) at human-like intervals. the second is to create a portal for webmasters that allows them special privileged access to analytics on data being extracted from their sites, and the ability to "turn…
Use a user-agent containing a URL to find out who and what you are, and honor my robots.txt. Having a panel for webmasters along with that would be fine.
Great suggestion... thanks for this one. We're putting this on our list