Portia, an open-source visual web scraper

From the video, I noticed that the HTML tags were also scraped in the large article text. Is there some way to remove those automatically? Or perform further processing?

Re: Portia, an open-source visual web scraper

#5

Post by compare » Tue, Apr 01, 2014, 2:36 PM UTC

Cool tool for developers, but since this one is open source, I think it opens up even more interesting possibilities for these tools to be integrated into part of a consumer app. Curation is the next big trend, right? I think I'll give that a try.

Re: Portia, an open-source visual web scraper

#6

Post by bsilvereagle » Tue, Apr 01, 2014, 2:37 PM UTC

I expected an April Fool's joke and found something pleasantly awesome and useful instead.

Source is here: https://github.com/scrapinghub/portia

Re: Portia, an open-source visual web scraper

#7

Post by esolyt » Tue, Apr 01, 2014, 2:40 PM UTC

Excellent. But the example presented in the video (scraping new articles) is a actually a case better solved with other technologies.

I imagine this will be useful when scraping sites like IMDB in case they don't have an API or their API is not useful enough.

Re: Portia, an open-source visual web scraper

#8

Post by pablohoffman » Tue, Apr 01, 2014, 2:46 PM UTC

rpedela wrote: post #4

From the video, I noticed that the HTML tags were also scraped in the large article text. Is there some way to remove those automatically? Or perform further processing?

Yes, you just need to select a different field type ("text", instead of "html").

Re: Portia, an open-source visual web scraper

#9

Post by pablohoffman » Tue, Apr 01, 2014, 2:50 PM UTC

rpedela wrote: post #2

Is there a live demo available?

Not yet.

Re: Portia, an open-source visual web scraper

#10

Post by alttab » Tue, Apr 01, 2014, 3:03 PM UTC

This is cool. Can I use it locally on internal sites too?