181–190 of 234 posts
This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.
thanks. glad you like it. pagination, dynamic tabs (and crawling in general) is a big feature we really want to add soon. a lot of people are asking for it. the challenge will be integrating it with the current UEX which we're trying to keep super simple.
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about doing with others. I was really excited to try this. I watched the demo video and it seemed straightforward. I went to try and use it on the demo page it provides, going through and adding things, but when I went to save it, I just received an er…
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).
Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are logged in.
Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.
Earlier quoted context omitted.
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ or http://hckrnews.com (no comments though).
I really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup? I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible. I can't tell you how often I click next page to find that I've taken too long…
I've used a bunch of HN skins that were supposedly better designed, but none of them stuck. Apparently it's just plain unnecessary for HN to be better.
Earlier quoted context omitted.
Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.
You could try doing IP address rotation by rotation EC2 instances or some other cloud services. I wrote a library for that. https://github.com/KrakeIO/resque-my-aws
Earlier quoted context omitted.
A lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.
Kimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.
http://edinburghhacklab.com/2013/09/probabalistic-scraping-o...
Earlier quoted context omitted.
I kind of just assumed that's what they were doing.
HTTPS is definitely a problem for proxy servers unless you the proxy server rewrites all the URLs in the html pages loaded as well as all the URLs of the Ajax calls to point back to the proxy server.
