so what do you do that import.io doesn't?
Show HN: Kimono – Never write a web scraper again
111–120 of 234 posts
Re: Show HN: Kimono – Never write a web scraper again
#112Earlier quoted context omitted.
The people who scrape data at Scraperwiki -- which was made by the same people who opened up parliamentary transcripts in the UK for the first time, and the UN's proceedings, and data about how MPs in London vote -- generally don't have an option to buy anything because the data's hidden by governments from the people who paid for it, on purpose. But by all means take this opportunity to dismiss all of us as freeload…
I have 17 scrapers on ScraperWIki classic for government data https://classic.scraperwiki.com/profiles/maxious/ It would have cost me $348/year to move those to new scraperwiki.
That reads as "Less than $1/day" to me...
Re: Show HN: Kimono – Never write a web scraper again
#113looks promising!
to be fully usable for me, there are some features missing:
- it lacks manual editing/correcting possibilities: i've tried to create an api for http://akas.imdb.com/calendar/?region=us with "date", "movie", "year". unfortunately, it failed to group the date (title) with the movies (list entries) but rather created two separate, unrelated collections (one for the dates, one for the movies).
- it lacks the ability to edit an api, the recommended way is to delete and recreate.
small bugreport: there was a problem saving the api, or at least i was told saving failed - it nevertheless seems to be stored stored in my account
Re: Show HN: Kimono – Never write a web scraper again
#114I like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy task to make accessible and reliable -- good luck!
It makes me wonder if there isn't a whole "API to web/mobile app with custom metadata" product in there somewhere. I can imagine a lot of folks starting to get into data analysis and pipelines having an easier time of it if they could just create a visual frontend in a few clicks.
Re: Show HN: Kimono – Never write a web scraper again
#115The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
It worked great for me (http://www.kimonolabs.com/kimonoapp/aws-status-check). It would be helpful if you shared what you tried to do when it failed. I don't work for or with Kimono, but am curious what does and does not work.
Re: Show HN: Kimono – Never write a web scraper again
#116Just use http://selectorgadget.com
Re: Show HN: Kimono – Never write a web scraper again
#117The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
In my case, worked for some pages and not for others. Currently I'm using Feedity: http://feedity.com for all business-centric data extraction and it has been working great (although not as flexible as kimono).
Re: Show HN: Kimono – Never write a web scraper again
#118Man that demo is impressive!
Re: Show HN: Kimono – Never write a web scraper again
#119Nice tool, slick UI. It worked for some pages and not for others. Currently I'm using Feedity: http://feedity.com for all business-centric data extraction and it has been working great (although not as flexible as kimono).
Re: Show HN: Kimono – Never write a web scraper again
#120Earlier quoted context omitted.
Use a user-agent containing a URL to find out who and what you are, and honor my robots.txt. Having a panel for webmasters along with that would be fine.
Great suggestion... thanks for this one. We're putting this on our list
Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.