Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Web scraping page analyzer

apify.com

Post reply on HN
Print view View original

1–10 of 52 posts

Re: Show HN: Web scraping page analyzer

#4

Post by jardah »

GSGSGS wrote: post #3
jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

Are you Jaroslov ? :)

Jaroslav, yes, I'm the author. Did you notice any problems or ways how I can improve it?

Re: Show HN: Web scraping page analyzer

#6

Post by _Chief »

jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

how about caching the default entry (static url instead) + attribs, to ease demoing. at the moment it's been analyzing for more than >5mins

Re: Show HN: Web scraping page analyzer

#7

Post by jancurn »

rmateus wrote: post #5
jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

Is it able to deal with digital certificates?

You can use https://www.htbridge.com/ssl/ for that

Re: Show HN: Web scraping page analyzer

#8

Post by jardah »

_Chief wrote: post #6
jardah wrote: post #2

I'm still testing it and improving it (there are so many different websites with different responses...), so If you have any comments I'm looking forward to what you think about it.

how about caching the default entry (static url instead) + attribs, to ease demoing. at the moment it's been analyzing for more than >5mins

Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo.

Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hanging runs or runs that ended with timeout.

Re: Show HN: Web scraping page analyzer

#9

Post by ComputerGuru »

jardah wrote: post #8
_Chief wrote: post #6

Earlier quoted context omitted.

how about caching the default entry (static url instead) + attribs, to ease demoing. at the moment it's been analyzing for more than >5mins

Good idea and I would implement that if I used an API from server to get the response. But currently I'm at the same time testing stability of Apify "Actor" solution and proxies, so for my case it's good that there are real requests with real responses, even if it's just from demo. Btw the fact that it's running for 5 minutes is a bug, that I will look at, since there is a timeout of 2 minutes and there are no hangin…

You also don’t want to get your server blocked by yelp if they do rate limiting.
Post reply on HN