Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

41–50 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#41

Post by hartard »

pranade wrote: post #32
phillmv wrote: post #29

The UX is great and a journalists everywhere will thank you. But outside of government websites I don't see how a lot of this is even legal, per se?

Thanks... yes, public data from governments is a great use case. Often a lot of apps built using scrapers will wind up driving up traffic/ sales a the source site so it's okay. We want to do responsible web scraping, so will respect webmasters robots.txt files to make sure it's legal.

I love the execution, but I also see inherent problems.

Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service.

You will encounter terms along the lines of:

"Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing: Use any robot, spider, scraper, other automatic device, or manual process to monitor, copy, or keep a database copy of the content or any portion of the Site."

Re: Show HN: Kimono – Never write a web scraper again

#43

Post by fnordfnordfnord »

pranade wrote: post #39
fnordfnordfnord wrote: post #36

>Sorry, can't kimonify >According that web site's data protection policy, we were unable to kimonify that particular page. Sigh... Oh well... Back to scraping.

What page were you trying to hit? We'll check it out

Pages buried in here: https://fannin4.wcjc.edu/

The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.

Re: Show HN: Kimono – Never write a web scraper again

#45

Post by eli »

phillmv wrote: post #29

The UX is great and a journalists everywhere will thank you. But outside of government websites I don't see how a lot of this is even legal, per se?

I'm not sure scraping itself is illegal, depending on what you're doing with the data. (Though it may be against a site's Terms of Use which may be binding. IANAL.)

I can tell you that on several occasions I've scraped commercial sites with the permission of the owner. They want me to have access to the data but don't have the time or ability to create a proper API.

Re: Show HN: Kimono – Never write a web scraper again

#46

Post by randomdrake »

The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward.

I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an error that something went wrong. Well, crap. That was a waste of time. Oh well, maybe it's just me.

Alright, I'll give it another shot using the website they used in the demo. Opened up a Hacker News discussion page and started to give it a try. Immediately it was far less intelligent than the demo. Clicking on a title proceeded to select basically every link on the page. Somehow I clicked on some empty spots as well. Nothing was being intelligently selected like it was in the demo. Fine, that wasn't working tremendously well, but I wanted to at least see the final result.

Same thing: just got an error that something went wrong and it couldn't save my work.

Disappointing. I still might try it again when it works 'cause it's a great idea if they really pulled it off. So far: doesn't seem to be the case.

Re: Show HN: Kimono – Never write a web scraper again

#48

Post by pranade »

guptaneil wrote: post #42

Nice work, this is much better than I expected! Does it require Chrome? It doesn't seem to work in Safari for me. Also, does Kimono work for scraping multiple pages or anything that requires authentication?

Great, it should work well on webkit browsers, what version of Safari are you using?

Re: Show HN: Kimono – Never write a web scraper again

#49

Post by guptaneil »

pranade wrote: post #48
guptaneil wrote: post #42

Nice work, this is much better than I expected! Does it require Chrome? It doesn't seem to work in Safari for me. Also, does Kimono work for scraping multiple pages or anything that requires authentication?

Great, it should work well on webkit browsers, what version of Safari are you using?

7.0.1, the latest. I also don't have Flash installed, but it doesn't look like you're using Flash. The entire top bar doesn't show for me. Feel free to email me and I can send you screenshots.

Re: Show HN: Kimono – Never write a web scraper again

#50

Post by pranade »

randomdrake wrote: post #46

The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…

Sorry you had a bad first experience. We've tested this on a lot of sites, and it works stably across a lot of different cases, but haven't solved it everywhere. Thanks for letting us know about the discussion page. We'll look into the bugs right now
Post reply on HN