161–170 of 234 posts
Earlier quoted context omitted.
Yeah, of course it won't work with HTTPS sites. They'd have to proxy those HTTPS sites and perform a MITM just to do it.
I kind of just assumed that's what they were doing.
This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.
https://chrome.google.com/webstore/detail/krakeio/ofncgcgajh...
Would love to meetup and exchange some ideas if you are based in Bay area.
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
Earlier quoted context omitted.
Please clarify: are you saying that right now you leave respecting robots.txt to the user?
At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt
I like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy task to make accessible and reliable -- good luck! It makes me wonder if there isn't a whole "API to web/mobile app with custom metadata" product in there somewhere. I can imagine a lot of folks starting to get into data analysis and pipelin…
Yes, we're excited about the possibilities of an end-to-end use case as well... in fact we were surprised when we found a lot of interest in front-end output layers on top of the APIs than just the APIs themselves. Would be curious to know what output features would be most valuable for you.
I'm doing a couple of data mining projects right now and simply being able to query and look into the API outputs, as well as my local database, without building a custom frontend would've saved me a bunch of time. But I'm thinking more of the knowledge worker, or even the power user who wants to view their Fitbit, Up and Lark data all in the same dashboard. Can't help but thinking this already exists somewhere though.
I don't know if it's just me or not, but it's not working for me in Firefox (OSX Mavericks 10.9.1 and Firefox v26). The X's and checkmarks aren't showing up next to the highlighted selections. Works fine in Safari.