Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrappy - Simple Perl Scraping Framework

ana.im

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Scrappy - Simple Perl Scraping Framework

#11

Post by [deleted] »

jonathansizz wrote: post #4
bravura wrote: post #2

At the very least, Scrappy (Perl) should link to Scrapy (Python). Otherwise, it seems remiss of this developer to pick a name that is easily confused that of an existing open-source project with similar purpose.

Oh, at the very least ! Then the module author should prostrate himself at the feet of the all-mighty Python community, trembling in unreserved awe whilst acknowledging that the world quite deservedly revolves around them ! Module authors of other (obviously inferior) languages, take note: always check if there's any Python code with a similar name before you choose a title for your project, to avoid embarrassment!

[deleted]

Re: Scrappy - Simple Perl Scraping Framework

#12

Post by bravura »

jonathansizz wrote: post #4
bravura wrote: post #2

At the very least, Scrappy (Perl) should link to Scrapy (Python). Otherwise, it seems remiss of this developer to pick a name that is easily confused that of an existing open-source project with similar purpose.

Oh, at the very least ! Then the module author should prostrate himself at the feet of the all-mighty Python community, trembling in unreserved awe whilst acknowledging that the world quite deservedly revolves around them ! Module authors of other (obviously inferior) languages, take note: always check if there's any Python code with a similar name before you choose a title for your project, to avoid embarrassment!

It's not about Python > Perl. It's about Scrapy is older (AFAIK) than Scrappy. If it were the other way around, I would have proposed that Scrapy rename or at least reference Scrappy.

The point is to avoid confusion among users who might not know about both tools.

Re: Scrappy - Simple Perl Scraping Framework

#13

Post by odonnell »

mwexler wrote: post #5

Most scrapers seem to break down on heavy dynamic/ajax pages. For example, anything made with GWT appears to provide little for the average scraper to grab (say, for automated daily tracking of android app downloads, for example). Short of reversing out the foreign pages api calls, has anyone encountered a solution to do more processing and then scrape the rendered page? (well, and short of using Selenium to script a…

Use Charles or another proxy and find what feed the page is loading, then parse that. At least two requests need to be made anyway.

Re: Scrappy - Simple Perl Scraping Framework

#14

Post by odonnell »

arkitaip wrote: post #6
mwexler wrote: post #5

Most scrapers seem to break down on heavy dynamic/ajax pages. For example, anything made with GWT appears to provide little for the average scraper to grab (say, for automated daily tracking of android app downloads, for example). Short of reversing out the foreign pages api calls, has anyone encountered a solution to do more processing and then scrape the rendered page? (well, and short of using Selenium to script a…

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.

DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.

Re: Scrappy - Simple Perl Scraping Framework

#15

Post by chsonnu »

arkitaip wrote: post #6
mwexler wrote: post #5

Most scrapers seem to break down on heavy dynamic/ajax pages. For example, anything made with GWT appears to provide little for the average scraper to grab (say, for automated daily tracking of android app downloads, for example). Short of reversing out the foreign pages api calls, has anyone encountered a solution to do more processing and then scrape the rendered page? (well, and short of using Selenium to script a…

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.

Have you tried Watir? I'm not sure if it'll solve your performance issue, but it's been at least twice as fast as Selenium for me.

Re: Scrappy - Simple Perl Scraping Framework

#16

Post by thirdusername »

odonnell wrote: post #14
arkitaip wrote: post #6

Earlier quoted context omitted.

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.

DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.

He's referring to this: http://en.wikipedia.org/wiki/Dhtml I'm not sure what DHTML you are thinking of that would be server-side.

Re: Scrappy - Simple Perl Scraping Framework

#17

Post by odonnell »

thirdusername wrote: post #16
odonnell wrote: post #14

Earlier quoted context omitted.

DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.

He's referring to this: http://en.wikipedia.org/wiki/Dhtml I'm not sure what DHTML you are thinking of that would be server-side.

Fuck, thinking of SHTML for some reason.
Post reply on HN