Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Portia, an open-source visual web scraper

blog.scrapinghub.com

Post reply on HN
Print view View original

11–20 of 70 posts

Re: Portia, an open-source visual web scraper

#11

Post by dabeeeenster »

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly...

All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

Re: Portia, an open-source visual web scraper

#12

Post by climatewarrior2 »

I've used Scrapy and it is the easiest and most powerful scraping tool I've used. This is so awesome. Since it is based on Scrapy I guess it should be possible to do the basic stuff with this tool and then take care of the nastier details directly on the code. I'll try it for my next scraping project.

Re: Portia, an open-source visual web scraper

#13

Post by agumonkey »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

It also depends on a coherent structure in HTML websites.

Domains running websites which are more like javascript frontend modules shouldn't be scraped at all, it screams for a public API.

Re: Portia, an open-source visual web scraper

#14

Post by yaph »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

Use the tools suitable for the task. There are a lot of those "simple" sites and I'm pretty sure a lot of people will stick to those "dated" methods of building sites, because search traffic still matters.

Re: Portia, an open-source visual web scraper

#16

Post by meritt »

emilsedgh wrote: post #15

I have a project which includes a huge list of websites which must be scraped heavily. My question is... Are these kind of tools suitable for 'heavy lifting', scraping hundreds of thousands of pages?

Yep. It's just a GUI that generates scrapy (python) code.

Re: Portia, an open-source visual web scraper

#17

Post by johndavi »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

The long tail is tough, but rules are useful when you only need to work with a small number of sites. And assuming, as you point out, less "modern" sites. (News sites tend to be mostly consistently manageable but, yes, smaller e-commerce players tend to adopt more modern techniques -- as befitting fashion-forward product lines, naturally).

Our (Diffbot) approach is to learn what news and product (and other) pages look like, and obviate the rules-management -- we also fully execute JS when rendering.

The web keeps evolving though, dang it. Tricky thing!

Re: Portia, an open-source visual web scraper

#18

Post by frabcus »

taskstrike wrote: post #3

Import.io, Kimono Labs, and now this. Web scraper -> data area is heating up.

It's always pretty hot! Some still around - like Kapow, Connotate, Mozenda, ScraperWiki (which I run). Some not - Needlebase.

Portia is more interesting because it is an open source scraping GUI - the GUIs tend to be very proprietary.

Post reply on HN