Live data from Hacker News

Welcome to the New AWS AI Blog

aws.amazon.com

91–98 of 98 posts

Re: Welcome to the New AWS AI Blog

#91

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default ( I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and…

checkout this blog on running predictions on Lambda with MXNet. It has pre-packaged numpy, openCV and MXNet.

https://aws.amazon.com/blogs/compute/seamlessly-scale-predic...

Re: Welcome to the New AWS AI Blog

#92
post #10

Earlier quoted context omitted.

What do you want to recognize? OCR is a better understood problem than a general neural net, so I think it's likely easier to improve its quality that to superseded the quality with image-based recognition.

Ideally, I would like to get all the information from the page. Phone numbers, who is suing whom, case caption, etc... With OCR, you get bits and pieces of information, but because I don't know what the type of the document is, it is difficult to determine where, structurally speaking, this information resides on a page. If I could use AI to determine the type of the doc, I would know the structure of the document an…

If your OCR is unable to recognize the characters on a full page, I don't think it when scanning a region either? Unless using full resolution of a full image is somehow too much for the algorithm in use. But then I'd just subdivide the entire image into regions, and scan them all independently. This is also a trivially parallelizable task, so you can throw many servers at it, if the time to get results is an issue.

Re: Welcome to the New AWS AI Blog

#93
post #32

Earlier quoted context omitted.

People keep recommending things and approaches, but I'm really not clear on what it is you're actually trying to solve, so most of the responses might not help. > Given that I have images and their types, could I "train" the AI with these million documents to recognize a new image that might come in? Do I understand this right: 1. You have lots of documents as images, and their type (brief, motion, court order) 2. Yo…

> Do I understand this right That is exactly right. As far as quality, if I classify the document wrong today (or fail to classify it), it's not that big of a deal to the system as a whole - but users will be "very" annoyed and have to correct it. > Are the documents typically very structurally different? Could I probably tell them apart without wearing glasses? Or are they largely the same, but with nuanced differen…

There's probably a way to determine the document type based on the differing structures (would have to see it to be sure). Alternately, is it not possible for a user to specify what type the document is?

Re: Welcome to the New AWS AI Blog

#94
post #6
post #5

Earlier quoted context omitted.

Principally, yes. However, the approach may be more nuisanced than that. If I were you, I would first pick a character recognition engine (which might have already been well trained) to convert the image to text. Once the text is there, that might serve as a better feature to classify the content. Furthermore, I had recommend converting words in the text to word-embeddings/ vectors using a suitable Glove or Word2Vec…

OCR is my current approach. I am not really happy with it. The quality of OCRing leaves much to be desired, probably due to the documents themselves being haphazardly handled by the court personnel. OCR itself is a pretty CPU intensive activity and takes a significant time to complete for many documents. Thus, I was looking for a more advanced approach.

> OCR itself is a pretty CPU intensive activity and takes a significant time to complete for many documents.

Leaving the quality part aside -- this job itself is easy to parallelize in that you can split it up by document or by page.

Open option is to run each job in Lambda asynchronously, with the input being a URL to the page or the full document, and have the job call back to you with the text of the page (or put it on S3 as a text file, or add it to a message queue, or whatever works). Regarding splitting: we've been using a python wrapper + pdfium for splitting PDFs into page images on Lambda, with excellent results.

To make the Lambda function, you'll either have to build e.g. Tesseract such that it fits into a 50MB zip, or download it while the Lambda function executes. LambCI has a set of docker containers that they've made for simulating lambda, and the "lambda:build" container makes building things easy and repeatable: https://github.com/lambci/docker-lambda. In a pinch, you can build on an Amazon Linux EC2 instance and it should work on Lambda, but you will have to be more careful about dynamic linking.

As another option: I'm not sure if it's been mentioned, but you can also try a ready-made OCR service before packaging up Tesseract, like this one: https://algorithmia.com/algorithms/ocr/SmartOCR.

So anyway, the performance part has good solutions, at least.

For fixing the accuracy: I know next to nothing about approximate string matching, but perhaps it would then be possible to do a fuzzy search over the text using something similar a Levenshtein automaton: https://en.wikipedia.org/wiki/Levenshtein_automaton.

You may also want to take a look at this: https://en.wikipedia.org/wiki/Bag-of-words_model

More broadly, I'm sure that there are text-based document classification methods that are robust against sloppy OCR. It may just take some research on the main approaches people take to document classification -- it's not my area, but my understanding is that this is typically approached with statistical methods. Otherwise your spam filter would get defeated by typos.

Re: Welcome to the New AWS AI Blog

#95
post #21

Earlier quoted context omitted.

I did see that, but that RSS feed (atom?) I don't think I've ever used that in my life. I'll Google and see what/how to use it. I see it everywhere though. edit: I did see that they have a podcast, nice, something to listen to while I walk 2 hours in the middle of the night.

Ughh, you're making me feel old. RSS is (was) an amazing system for tracking website updates over time. It lost out to Twitter and Facebook (which don't remotely replace the use case), though it is still what makes podcasts work. I highly recommend Inoreader as a great RSS reader.

Thanks for the recommendation. I tried one, I am not quite sure how it works yet at least for that AWS one. One page was a full-article, another page was several articles in one, it was odd but thank you.

Regarding Blogger back in like 2011 or so, I saw RSS feed all the time at the bottom (usually still do).

Re: Welcome to the New AWS AI Blog

#96
post #21

Earlier quoted context omitted.

I did see that, but that RSS feed (atom?) I don't think I've ever used that in my life. I'll Google and see what/how to use it. I see it everywhere though. edit: I did see that they have a podcast, nice, something to listen to while I walk 2 hours in the middle of the night.

Try feedly.

This is the one I tried due to editor choice/most downloads. I'm not sure if my experience was due to the source (AWS) or feedly's design. Guess I would just have to get used to it.

What I want is offline daily download of specific stuff. Being lazy myself, to build scrapers and download them to my phone but not an Android dev only web dev at this stage.

Re: Welcome to the New AWS AI Blog

#97

Earlier quoted context omitted.

It sounds like you're stretching to make the use case fit inside a Lambda. Have you considered extracting the assessment functionality into an SWF activity, invoked from your Lambda via StepFunctions?

No, we haven't looked at SWF at all - might do some reading up on that. We considered running it on EC2, but the economics just didn't work out for our needs (hundreds of parallel processed jobs with irregular spikes, <5s runtimes per invocation and some others)

You might also consider AWS batch. It's designed for this type of use case.

Re: Welcome to the New AWS AI Blog

#98
post #95

Earlier quoted context omitted.

Ughh, you're making me feel old. RSS is (was) an amazing system for tracking website updates over time. It lost out to Twitter and Facebook (which don't remotely replace the use case), though it is still what makes podcasts work. I highly recommend Inoreader as a great RSS reader.

Thanks for the recommendation. I tried one, I am not quite sure how it works yet at least for that AWS one. One page was a full-article, another page was several articles in one, it was odd but thank you. Regarding Blogger back in like 2011 or so, I saw RSS feed all the time at the bottom (usually still do).

Running a partial text RSS feed through here will often replace it with a full text RSS feed: http://fivefilters.org/content-only/
Post reply on HN