41–50 of 52 posts
Nice work! It seems that it only supports microdata and not RDFa at the moment?
Yep only microdata. I completely forgot about RDFa. I'm immediately writing RDFa to my todo list. It would be a great addition.
Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?
I'd have money on using Horseman with phantomjs in node.
Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?
This tool basicaly performs the simplest data loading, it opens the webpage, then waits till most xhr requests are done, wait's a second (tio give JS time to manipulate DOM) and then loads data from the page. This way, it has what user sees when he opens the page in browser. So if the data is visible, or loaded through XHR or hidden in global JS variable it will see it. For more advanced usage (like clicking, or subm…
Earlier quoted context omitted.
Yes, that is probably the problem, when I looked for the text it returned: [ 0:{ "selector":".bloc-blanc > p:nth-child(1)" "text":" 0 école(s) correspondent à votre recherche " } ]
Aha! I see, it shows data based on POST request from FORM on this page http://www.dsden93.ac-creteil.fr/spip/spip.php?page=annu1d so if you provide just a link to the results page without the POST data then it will show you nothing. Sadly the tool currently does not allow for sending POST requests to the websites.
Edit: the page changed and it's not working anymore. Sorry for the false alarm, my bad.
* shameless plug * Our little startup, Feedity - https://feedity.com, helps create custom RSS feeds for any webpage, utilizing Chrome for full-rendering and many other tweaks & techniques under the hood for seamless & scalable indexing.
Earlier quoted context omitted.
Not that i can see from a surface view, i think documentation can be improved :). Personally like the idea of APIFY, saw it a few months ago. Are you guys hiring ? :D
Yes we are! Please see https://www.apify.com/jobs
I just sent an application for the Junior Web Developer position.
Looking forward to hearing back!
Looks awesome! Does the tool work when trying to access websites behind web application firewalls? eg. F5 WAF [1] https://f5.com/glossary/web-application-firewall
Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?
I'd have money on using Horseman with phantomjs in node.