Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Welcome to the New AWS AI Blog

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 98 posts

Re: Welcome to the New AWS AI Blog

#11

Post by niklasrde »

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default (I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and Services.

This could be as simple as publicly documenting the time lambda's stay 'warm' for and retain data in /tmp persisting through multiple invocations or some other examples on how an AI workflow could be implemented with popular custom engines such as TensorFlow.

Does anybody else have any experience in this regard?

Re: Welcome to the New AWS AI Blog

#13

Post by barrkel »

starik36 wrote: post #6
bhaavan wrote: post #5

Earlier quoted context omitted.

Principally, yes. However, the approach may be more nuisanced than that. If I were you, I would first pick a character recognition engine (which might have already been well trained) to convert the image to text. Once the text is there, that might serve as a better feature to classify the content. Furthermore, I had recommend converting words in the text to word-embeddings/ vectors using a suitable Glove or Word2Vec…

OCR is my current approach. I am not really happy with it. The quality of OCRing leaves much to be desired, probably due to the documents themselves being haphazardly handled by the court personnel. OCR itself is a pretty CPU intensive activity and takes a significant time to complete for many documents. Thus, I was looking for a more advanced approach.

I think you'd have a better time doing your OCR on AWS - spread the CPU intensive activity across multiple machines. Even if the resulting text has errors, if you're doing further document classification, it would be better to do it on the text including the errors, than on the original images.

Re: Welcome to the New AWS AI Blog

#16

Post by skewart »

starik36 wrote: post #2

I am a complete noob to the AI space but I was wondering whether the following is possible (in AWS). I have a million scanned images of court documents. Some are briefs, some are motions, some are court orders, etc... Given that I have images and their types, could I "train" the AI with these million documents to recognize a new image that might come in?

AWS doesn't offer any high-level services for training your own custom model. You'd have to build the neural network yourself and deploy EC2 boxes to run it.

I've had success with Clarifai's [0] custom CV model API in the past. You basically upload batches of labeled images to train a model, and then you can submit new images for classification.

Of course, I have no idea how effective it would be for your documents. Obviously it depends on how visually distinct the different types are.

[0] https://www.clarifai.com

Re: Welcome to the New AWS AI Blog

#17

Post by techdragon »

I find it frustrating for all the power they want to give me... that some basic service design is lacking.

Polly is a stand alone component but the reverse is closely bound up into Lex which is a conversational interface API.

Amazon has internally built an engine I could ask to convert an audio file in S3 into a text content representative of the audio file... yet I can only use Lex to drive a conversation via text and audio.

If AWS really want to give me the power of their AI tools. How about unbundling them?

Re: Welcome to the New AWS AI Blog

#18

Post by perfmode »

starik36 wrote: post #6
bhaavan wrote: post #5

Earlier quoted context omitted.

Principally, yes. However, the approach may be more nuisanced than that. If I were you, I would first pick a character recognition engine (which might have already been well trained) to convert the image to text. Once the text is there, that might serve as a better feature to classify the content. Furthermore, I had recommend converting words in the text to word-embeddings/ vectors using a suitable Glove or Word2Vec…

OCR is my current approach. I am not really happy with it. The quality of OCRing leaves much to be desired, probably due to the documents themselves being haphazardly handled by the court personnel. OCR itself is a pretty CPU intensive activity and takes a significant time to complete for many documents. Thus, I was looking for a more advanced approach.

You don't need perfect character recognition. It's just gotta be good enough. The way you determine good enough is by completing the pipeline and measuring the result.

Re: Welcome to the New AWS AI Blog

#19

Post by sandusk »

niklasrde wrote: post #11

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default ( I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and…

I'm making over $7k a month working part time. I kept hearing other people tell me how much money they can make online so I decided to look into it. Well, it was all true and has totally changed my life. This is what I do... http://bit.ly/2atnA1a

Re: Welcome to the New AWS AI Blog

#20

Post by allenz »

starik36 wrote: post #7
aerovistae wrote: post #4

Earlier quoted context omitted.

Is that possible with AI? Yes, almost trivially. Is that possible with AWS AI? > Finally, we provide AI engines, a collection of open-source, deep learning frameworks for academics and data scientists who want to build cutting edge, sophisticated intelligent systems, pre-installed configured on a convenient machine image. Which is to say anything that TensorFlow can do, AWS AI can do. So yes. But is it possible witho…

> almost trivially If not AWS, how could I get started elsewhere? Every tutorial I've looked at deals with a CVS data file, not an image. Could you point me to a resource that has an example of how to train AI with image documents?

You would just use pixel values as inputs, see https://www.tensorflow.org/get_started/mnist/beginners

You can make it more performant by downsizing the documents, but obviously this is incredibly GPU-intensive to train. I second the suggestions to just use OCR.

Post reply on HN