Well done on the product & solving a clear need! This is extremely useful for hackathons/prototyping. I also loved the live demo in the blog post and you did a wonderful job with the design/layout/colorscheme of the site.
Show HN: Kimono – Never write a web scraper again
21–30 of 234 posts
Re: Show HN: Kimono – Never write a web scraper again
#22This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny.
Feature proposal: deal with pagination.
Re: Show HN: Kimono – Never write a web scraper again
#23This looks really slick. What happens if a website you're scraping changes its design? Do you respect robots.txt?
If it changes the format significantly, the scraper will break, so for now you'll have to use the tool to rebuild. You will see on your API status page that it's down. As for robots.txt, we do respect it... for now we're leaving that to the user, but we're trying to implement a proactive way of checking for disallows and stopping those scrapers from being built.
Re: Show HN: Kimono – Never write a web scraper again
#24How (if at all) does this run on javascript heavy sites?
Re: Show HN: Kimono – Never write a web scraper again
#25Looks awesome, however I keep getting errors and 404s. Could this be an issue on my end (seems to be working for others) or just HN making the servers beg for mercy?
Where are you getting the 404s? We will check into it now
Re: Show HN: Kimono – Never write a web scraper again
#26How (if at all) does this run on javascript heavy sites?
JS-heavy sites can be tricky. We position it so it should execute after most of the on-page JS, so it handles a lot of cases. There are still sites that break it though.... we're trying to tackle these guys one by one right now, as we try to generalize a broader solution
Re: Show HN: Kimono – Never write a web scraper again
#27Does it do logging in to websites then fetching? Do you plan to add scripting to it?
We don't support logging in yet, but it's a feature we're working on adding. Scripting will also be cool, but it's right now further down our feature queue
Re: Show HN: Kimono – Never write a web scraper again
#28Very nice job. What about scraping data from password-protected pages?
Re: Show HN: Kimono – Never write a web scraper again
#29The UX is great and a journalists everywhere will thank you.
But outside of government websites I don't see how a lot of this is even legal, per se?
Re: Show HN: Kimono – Never write a web scraper again
#30Very cool, and I like that the link is your announcement page running inside of the demo. Really drives home the idea. That said, it looks like it can't do media right now. I would love it if it could at least give me a url for images/other media.
It's a great suggestion, thanks! ... image extraction would be cool, and it's on our shortlist of features to build next