Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

Post reply on HN
Print view View original

141–150 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#144

Post by pranade »

thatthatis wrote: post #124
pranade wrote: post #23

Earlier quoted context omitted.

If it changes the format significantly, the scraper will break, so for now you'll have to use the tool to rebuild. You will see on your API status page that it's down. As for robots.txt, we do respect it... for now we're leaving that to the user, but we're trying to implement a proactive way of checking for disallows and stopping those scrapers from being built.

Please clarify: are you saying that right now you leave respecting robots.txt to the user?

At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt

Re: Show HN: Kimono – Never write a web scraper again

#145

Post by GBond »

prolways wrote: post #123
roryokane wrote: post #107

Earlier quoted context omitted.

To me, the favicon merely looks like a sumo wrestler’s head with a short ponytail and scowling/serious eyebrows. I can’t tell what NSFW thing you see it as.

Until I read this thread I also saw a sumo or an angry onion, but I believe the picture is actually a person facing away from us undoing their kimono.

You see all that in a 32x32 pixel image?

Re: Show HN: Kimono – Never write a web scraper again

#147

Post by pranade »

thatthatis wrote: post #143

This is my third time trying to get an answer to this question: does your crawler automatically respect robots.txt?

So sorry for missing this earlier. See our response in comments below: "At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt"

Re: Show HN: Kimono – Never write a web scraper again

#148

Post by thinkzig »

Great work so far. The tool was very intuitive and easy to use.

My suggestion: once I've defined an API, let me apply it to multiple targets that I supply to you programatically.

The use case driving my suggestion: I'm an affiliate for a given eCommerce site. As an affiliate, I get a data feed of items available for sale on the site, but the feed only contains a limited amount of information. I'd like to make the data on my affiliate page richer with extra data that I scrape from a given product page that I get from the feed.

In this case, the page layout for all the various products for sale is exactly the same, but there are thousands of products.

So I'd like to be able to define my Kimono API once - lets call it CompanyX.com Product Page API - then use the feed from my affiliate partner to generate a list of target URLs that I feed to Kimono.

Bonus points: the list of products changes all the time. New products are added, some go away, etc. I'd need to be able to add/remove target URLs from my Kimono API individually as well as adding them in bulk.

Thanks for listening. Great work, again. I can't wait to see where you go with this.

Cheers!

Re: Show HN: Kimono – Never write a web scraper again

#149

Post by the_french »

chii wrote: post #132
pranade wrote: post #87

Earlier quoted context omitted.

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

how are you going to do it without having to know the actual authentication key(s)? if i don't trust anyone enough to give my auth away, and so unless the site being scraped has some sort of oauth support, how are you going to get any data? of course, if this was an offline product, or self-hosted product, then it would solve that problem of auth instantly.

Would there any way to fake the beginning of an OAuth session with Facebook, Google or any other OAuth authenticated site? Kind of like replaying cookies to hijack sessions?
Post reply on HN