Scrapely: The brains behind Portia, our visual web scraping tool • Hacker News Board

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#11

Post by unsettledtck » Thu, Jul 07, 2016, 4:54 PM UTC

rkrzr wrote: post #9

This looks like a good solution for scrapers where "close enough" is good enough. If you need 100% accuracy you can always fall back to using scrapy directly and make your scraping logic as accurate as you need. But in many cases you can live with some false positives and then this tool looks like it will fit the bill.

We've actually developed a way that you can convert Portia projects into Scrapy spiders: https://blog.scrapinghub.com/2016/06/29/introducing-portia2c...

and since this is all open source, here's a link to GitHub: https://github.com/scrapinghub/portia2code

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#12

Post by rlndmx » Thu, Jul 07, 2016, 5:44 PM UTC

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

Here is a '12 survey: http://arxiv.org/abs/1207.0246

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#13

Post by abc03 » Thu, Jul 07, 2016, 8:39 PM UTC

kmike84 wrote: post #10

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

There are many methods for different tasks. What do you mean by 'data extraction', do you have some specific examples?

After the OCR of documents, I have used mostly regex to extract information from semi-structured documents. One example would be invoices (invoice number, total amount etc.), another would be to extract product names, SKU numbers etc. from various documents.

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#14

Post by ahljoh » Fri, Jul 08, 2016, 11:18 AM UTC

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

We would need more context/information about your specific objectives.

- document conversion (pdftotext, pdfbox, apache tabula, etc.)

- OCR (tesseract, pypdfocr, etc.)

- Named-Entity-Recognition (NER) i.e. finding and recognizing entities in text (DBPedia Spotlight, stanford NER via NLTK, spacy)

- coreference resolution, dependency parsing (spacy, syntaxnet)

Re: Scrapely: The brains behind Portia, our visual web scraping tool

#15

Post by abc03 » Fri, Jul 08, 2016, 12:44 PM UTC

ahljoh wrote: post #14

abc03 wrote: post #7

Does anyone know what methods are state of the art in machine learning for data extraction in general or where I could get an overview (invoices,images, documents etc.)?

We would need more context/information about your specific objectives. - document conversion (pdftotext, pdfbox, apache tabula, etc.) - OCR (tesseract, pypdfocr, etc.) - Named-Entity-Recognition (NER) i.e. finding and recognizing entities in text (DBPedia Spotlight, stanford NER via NLTK, spacy) - coreference resolution, dependency parsing (spacy, syntaxnet)

Thanks. Some great keywords to investigate. I'm namely interested in two areas at the moment: - invoices (I guess NER would be partially an Option) - web scrapping (wrapper induction)