Scrapely: The brains behind Portia, our visual web scraping tool

#1

Post by unsettledtck » Thu, Jul 07, 2016, 1:20 PM UTC

Scrapely: The brains behind Portia, our visual web scraping tool
blog.scrapinghub.com

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#2

Post by merraksh » Thu, Jul 07, 2016, 1:46 PM UTC

I wonder if this was submitted after reading today's submission (https://news.ycombinator.com/item?id=12047234) about (real) Portia spiders and googling the subject.

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#3

Post by unsettledtck » Thu, Jul 07, 2016, 2:00 PM UTC

merraksh wrote: post #2

I wonder if this was submitted after reading today's submission ( https://news.ycombinator.com/item?id=12047234 ) about (real) Portia spiders and googling the subject.

We did indeed model our Portia on the real spiders. We'd like to think our version is a wee bit cuter...

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#4

Post by david-given » Thu, Jul 07, 2016, 2:34 PM UTC

unsettledtck wrote: post #3

merraksh wrote: post #2

I wonder if this was submitted after reading today's submission ( https://news.ycombinator.com/item?id=12047234 ) about (real) Portia spiders and googling the subject.

We did indeed model our Portia on the real spiders. We'd like to think our version is a wee bit cuter...

After finding this picture of a real Portia, I have to disagree.

https://c1.staticflickr.com/7/6096/6306406141_3b237e21ee_b.j...

Look at those big, soulful black eyes...

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#5

Post by pavel_lishin » Thu, Jul 07, 2016, 2:42 PM UTC

merraksh wrote: post #2

I wonder if this was submitted after reading today's submission ( https://news.ycombinator.com/item?id=12047234 ) about (real) Portia spiders and googling the subject.

I wonder if that submission was in any way related to Peter Watts' novel "Echopraxia"...

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#6

Post by jsargiox » Thu, Jul 07, 2016, 2:47 PM UTC

merraksh wrote: post #2

I wonder if this was submitted after reading today's submission ( https://news.ycombinator.com/item?id=12047234 ) about (real) Portia spiders and googling the subject.

I didn't know there was a real "portia" spider... the things you learn everyday

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#7

Post by abc03 » Thu, Jul 07, 2016, 2:47 PM UTC

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#8

Post by Buttons840 » Thu, Jul 07, 2016, 3:02 PM UTC

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

Research papers often use the phrase "wrapper induction" when discussing automatic data extraction. Once I discovered that phrase I found several papers. I'm on mobile so can't link any.

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#9

Post by rkrzr » Thu, Jul 07, 2016, 3:58 PM UTC

This looks like a good solution for scrapers where "close enough" is good enough. If you need 100% accuracy you can always fall back to using scrapy directly and make your scraping logic as accurate as you need. But in many cases you can live with some false positives and then this tool looks like it will fit the bill.

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#10

Post by kmike84 » Thu, Jul 07, 2016, 4:34 PM UTC

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

There are many methods for different tasks. What do you mean by 'data extraction', do you have some specific examples?