Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Web scraping page analyzer

apify.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Show HN: Web scraping page analyzer

#21

Post by bpicolo »

ComputerGuru wrote: post #9
jardah wrote: post #8

Earlier quoted context omitted.

Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo. Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hangin…

You also don’t want to get your server blocked by yelp if they do rate limiting.

They have fairly aggressive scraper detection (and this is also against their ToS)

Re: Show HN: Web scraping page analyzer

#22

Post by lanewinfield »

jardah wrote: post #13
BrandoElFollito wrote: post #12

Is there a way to access an authenticated web site?

Sadly not for now. Our company has a solution for that (for some websites), but currently this tool does not have this functionality, since I wanted it be as simple as possible. Maybe in the future.

Authentication support on this would make it an instant purchase for me.

Re: Show HN: Web scraping page analyzer

#23

Post by jancurn »

GSGSGS wrote: post #15
jardah wrote: post #4

Earlier quoted context omitted.

Jaroslav, yes, I'm the author. Did you notice any problems or ways how I can improve it?

Not that i can see from a surface view, i think documentation can be improved :). Personally like the idea of APIFY, saw it a few months ago. Are you guys hiring ? :D

Yes we are! Please see https://www.apify.com/jobs

Re: Show HN: Web scraping page analyzer

#24

Post by lozzo »

lanewinfield wrote: post #22
jardah wrote: post #13

Earlier quoted context omitted.

Sadly not for now. Our company has a solution for that (for some websites), but currently this tool does not have this functionality, since I wanted it be as simple as possible. Maybe in the future.

Authentication support on this would make it an instant purchase for me.

why ? and how would you envisage authentication support to work ? (p.s. awesome tool guy...the clever part is to allow user the ability to describe their process via JavaScript. Hence a perfect DSL (domain specific language))

Re: Show HN: Web scraping page analyzer

#26

Post by JustARandomGuy »

jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

Suppose I wanted to extract an image that gets loaded async via Javascript (For example, a Pinterest page). How would that work? Looking at your documentation, it looks like I could parse the XHR array you supply. Could you suggest any other ways? I'm calling out Pinterest as an example here because they try to block their images from being easily downloaded, but if you have any other examples I'd like to hear them.

It would be great if the page analyzer could supply a list of all the assets loaded with the web page; for example, any asset with a media type of image/* is listed in an images array, and so forth.

Re: Show HN: Web scraping page analyzer

#27

Post by jardah »

ComputerGuru wrote: post #9
jardah wrote: post #8

Earlier quoted context omitted.

Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo. Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hangin…

You also don’t want to get your server blocked by yelp if they do rate limiting.

It's why I'm using proxies, every request is routed through different proxy address and the application as whole is rate limited. So hopefully I'm not making too much traffic on yelp. They are just a perfect example because they are using all types of data I'm looking for. When I find more good examples I will add them and rotate them for every page load.

Btw when it comes to ToS and scraping, this is not much different from accessing their website through normal browser only instead of rendered content we should you analyzed data. The page is only loaded once same as in browser.

Re: Show HN: Web scraping page analyzer

#28

Post by jardah »

lanewinfield wrote: post #22
jardah wrote: post #13

Earlier quoted context omitted.

Sadly not for now. Our company has a solution for that (for some websites), but currently this tool does not have this functionality, since I wanted it be as simple as possible. Maybe in the future.

Authentication support on this would make it an instant purchase for me.

Some general authentication (like separate input fields for your login credentials on the website) could be potentialy done (but very unsafe for user of the tool, since you would be sending us your credentials as plaintext). But authentication as whole is sadly not as general as semantic data on the web. Not every website has the same login form(different fields), some use captchas, some use authenticators, some do robot checking for too fast logins.

Re: Show HN: Web scraping page analyzer

#29

Post by jardah »

at_smith wrote: post #19

Awesome tool! How do you handle scraping data that's hiding behind layers of ~fancy~ JS libraries? Is it as simple as triggering click events, pausing for loading, and then grabbing the information?

This tool basicaly performs the simplest data loading, it opens the webpage, then waits till most xhr requests are done, wait's a second (tio give JS time to manipulate DOM) and then loads data from the page. This way, it has what user sees when he opens the page in browser. So if the data is visible, or loaded through XHR or hidden in global JS variable it will see it.

For more advanced usage (like clicking, or submiting a search request) it would need to have some kind of scenario like: "Click on this" -> "wait till this loads" -> "type something here" -> "scroll to this" -> load data.

Which is possible with headless chrome, so the trick is to make it general and easy to use (something like recording what user does through chrome plugin). Maybe in future versions :)

Re: Show HN: Web scraping page analyzer

#30

Post by jardah »

JustARandomGuy wrote: post #26
jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

Suppose I wanted to extract an image that gets loaded async via Javascript (For example, a Pinterest page). How would that work? Looking at your documentation, it looks like I could parse the XHR array you supply. Could you suggest any other ways? I'm calling out Pinterest as an example here because they try to block their images from being easily downloaded, but if you have any other examples I'd like to hear them.…

Actually the list of assets shouldn't be that hard. Looking at pinterest the xhr requests for images are loaded immediately when page is open, so potentialy it then it's catched in onRequest function (only now I'm aborting the requests to save network trafic). I will try it our tomorrow and let you know in comment.

Also, looking at pinterest, it's server rendered through ReactJS, so there is #initial-state script tag with first few images preloaded as urls, so if you cared only about the images on top without scrolling then this is the safest bet.

Post reply on HN