Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Web scraping page analyzer

apify.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Show HN: Web scraping page analyzer

#31

Post by jardah »

GSGSGS wrote: post #15
jardah wrote: post #4

Earlier quoted context omitted.

Jaroslav, yes, I'm the author. Did you notice any problems or ways how I can improve it?

Not that i can see from a surface view, i think documentation can be improved :). Personally like the idea of APIFY, saw it a few months ago. Are you guys hiring ? :D

:D yep the documentation needs a lot of work. It started as a test of an idea, then slowly became a usable tool and the code was getting incrementaly more complex without me event noticing. I only added the readme on github yesterday and there are basicaly no tests... :(

Re: Show HN: Web scraping page analyzer

#35

Post by jardah »

guilamu wrote: post #34

Not giving me anything useful on this pretty straightforward table: http://www.dsden93.ac-creteil.fr/spip/spip.php?page=ecoles

When I open the link in my browser it shows "0 école(s) correspondent à votre recherche" and no table, probably what happens to the analyzer too.

Re: Show HN: Web scraping page analyzer

#36

Post by jardah »

guilamu wrote: post #34

Not giving me anything useful on this pretty straightforward table: http://www.dsden93.ac-creteil.fr/spip/spip.php?page=ecoles

Yes, that is probably the problem, when I looked for the text it returned:

[ 0:{ "selector":".bloc-blanc > p:nth-child(1)" "text":" 0 école(s) correspondent à votre recherche " } ]

Re: Show HN: Web scraping page analyzer

#37

Post by jardah »

jardah wrote: post #36
guilamu wrote: post #34

Not giving me anything useful on this pretty straightforward table: http://www.dsden93.ac-creteil.fr/spip/spip.php?page=ecoles

Yes, that is probably the problem, when I looked for the text it returned: [ 0:{ "selector":".bloc-blanc > p:nth-child(1)" "text":" 0 école(s) correspondent à votre recherche " } ]

Aha! I see, it shows data based on POST request from FORM on this page http://www.dsden93.ac-creteil.fr/spip/spip.php?page=annu1d so if you provide just a link to the results page without the POST data then it will show you nothing. Sadly the tool currently does not allow for sending POST requests to the websites.

Re: Show HN: Web scraping page analyzer

#38

Post by jardah »

Just a quick update: Thank you for using it and playing around with it. Looking at the usage and results I found a quite a lot of things to improve. Which is great, since it's hard to develop something like this without real usage data.

Re: Show HN: Web scraping page analyzer

#39

Post by cseelus »

jardah wrote: post #29
at_smith wrote: post #19

Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?

This tool basicaly performs the simplest data loading, it opens the webpage, then waits till most xhr requests are done, wait's a second (tio give JS time to manipulate DOM) and then loads data from the page. This way, it has what user sees when he opens the page in browser. So if the data is visible, or loaded through XHR or hidden in global JS variable it will see it. For more advanced usage (like clicking, or subm…

Could be an interesting enhancement. Sounds a little bit like what Capybara, a test framework for Ruby apps can do[1], things like

  click_link('Link Text')
  fill_in('Password', with: 'Seekrit')
  choose('A Radio Button')
  check('A Checkbox')
  uncheck('Another Checkbox')
  select('Option', from: 'Select Box')
1) https://github.com/teamcapybara/capybara#navigating
Post reply on HN