Live data from Hacker News

Show HN: Kimono – Never write a web scraper again

kimonify.kimonolabs.com

81–90 of 234 posts

Re: Show HN: Kimono – Never write a web scraper again

#81
post #72

Thank you for building a tool I been wanting so I don't have to! Can't wait to play around with this tonight. Suggestion. Allow one to select images.

Great add. It's on our shortlist - a popular request!

Also, the ability to style it myself would be nice :)

Re: Show HN: Kimono – Never write a web scraper again

#82
Like the parameter passthrough feature. Take a look at places where the parameters are part of the URL structure. For example a Target product page http://www.target.com/p/men-s-c9-by-champion-impact-athletic...

In order to get data for a different product, I will have to modify the URL itself. I think same holds true for blog posts.

Re: Show HN: Kimono – Never write a web scraper again

#84
post #76
post #73

Constructive Tone: I figured that it might be nifty to scrape cedar pollen count information from a calendar and then shoot myself an email when it was higher than 100 gr/m3. This would be a pretty difficult thing to grab when scraping normally, but the app errors before loading the content: https://www.keepandshare.com/calendar/show_month.php?i=19409... JS error: An error occurred while accessing the server, please…

Thanks for letting us know. Just tried and am getting the same error. The page is loading content dynamically from another source... We'll look into this and see if we can get it working on this page

Do you support POSTs for fetching dynamic data? I found where it's pulling from, here's the curl command:

curl "https://www.keepandshare.com/calendar/fns_asynch_api.php?r=0... --data "action=getrange&i=1940971&from=2013-12-26&to=2014-02-06"

Re: Show HN: Kimono – Never write a web scraper again

#85
That UI made me go wow, this could be an awesome tool. Idea that pops into my mind is being able to grab data from those basic local sites run by councils, local news papers etc and putting it into a useful app.

How dedicated are you guys to making this work because I'd imagine there are quite a few technical hurdles in keeping a service like this working long term while not getting blocked by various sites?

Re: Show HN: Kimono – Never write a web scraper again

#86

Like the parameter passthrough feature. Take a look at places where the parameters are part of the URL structure. For example a Target product page http://www.target.com/p/men-s-c9-by-champion-impact-athletic... In order to get data for a different product, I will have to modify the URL itself. I think same holds true for blog posts.

Yes, it's a great point. We're working on updating the query param passthrough to handle params within the URL structure.

Re: Show HN: Kimono – Never write a web scraper again

#87

Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.

Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.

Re: Show HN: Kimono – Never write a web scraper again

#89
post #80

really excited to see this. i've had the idea (and nearly this execution) in mind for years but no use or ambition to get it done. given the pricing though i'm almost motivated to make my own. as a hosted service the fees make sense with the offerings. but not only would i rather host my own- it would be cheaper all around. would you consider adding a free or cheap self hosted option? aside, i think there is a mislab…

Yes, we're in beta right now, while we're still working out the bugs. For beta, it's free for 30 APIs.

thanks for the response! any chance at a self hosted option? i'd even still pay (once) for a self hosted version.

Re: Show HN: Kimono – Never write a web scraper again

#90
as someone building a home grown proprietary scraping engine. Consider alternative locations of elements. Most sites are using templating engines so its fairly reliable to find things in the same place, but more often than you might expect, things move a round ever so slightly. Navigation is a fun one also. ;)
Post reply on HN