Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer preview of TensorFlow Lite

developers.googleblog.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Developer preview of TensorFlow Lite

#71

Post by fiatjaf »

dgacmu wrote: post #45
fiatjaf wrote: post #25

So we'll start to see more and more battery-consuming "AI" apps in mobile devices?

And we'll start to see more battery-efficient hardware to run those apps without consuming all of your battery. :) (I'm saying that glibly, but I'm dead serious -- look at what we've seen emerge just this year in Apple's Neural Engine, the Pixel Visual Core, rumored chips from Qualcomm, and the Movidius Myriad 2. The datacenter was the first place to get dedicated DNN accelerators in the form of Google's TPU, but the…

Thank you.

Re: Developer preview of TensorFlow Lite

#72

Post by sandGorgon »

rajatmonga wrote: post #68
sandGorgon wrote: post #52

Earlier quoted context omitted.

Hi, I have written about this before ( https://news.ycombinator.com/item?id=15595689 ) , but are there serialization fixes between cloud training and mobile ? We have had huge issues in trying to figure out how to save models (freeze graph,etc) and load it on Android. If you look at my previous thread - it also mentions bugs,threads and support requests where people are consistently confused.

Agree, that is a big problem that we are working hard to solve. It isn't solved in this release, but it is high up on our task list.

hey, thanks for the reply!

petewarden (https://news.ycombinator.com/item?id=15596990) from Google is also working on this - so im really hopeful you guys will have something soon. This is a serious blocker for doing anything reasonable in TF.

Re: Developer preview of TensorFlow Lite

#74

Post by infnorm »

restricted_ptr wrote: post #56
petewarden wrote: post #34

Earlier quoted context omitted.

One nice thing about Lite is that it's a lot easier to just include the operations you need (compared to TensorFlow 'classic'), there's fusion for common patterns, and the base interpreter is only 70KB. That covers a lot of the advantages of using XLA for mobile apps. In return you have the ability to load models separately from the code, and the ops are hand-optimized for ARM. I'm still a fan of XLA, and I expect th…

How about quantization? Does tensorflow lite perform quantization or is it tensorflow supposed to do it? Is it iterative process or straightforward? Or are you training quantized models as nn api docs say?

The quantization is done with a special training script that is quantization aware. We will be open sourcing a mobilenet quantized training script to show how to do this soon.

Re: Developer preview of TensorFlow Lite

#75

Post by infnorm »

dangjc wrote: post #70
rajatmonga wrote: post #2

TensorFlow Lite is TensorFlow’s lightweight solution for mobile and embedded devices! TensorFlow has always run on many platforms, from racks of servers to tiny devices, but as the adoption of machine learning models has grown over the last few years, so has the need to deploy them on mobile and embedded devices. TensorFlow Lite enables low-latency inference of on-device machine learning models. Looking forward to yo…

Is it lite enough to compile with Emscripten and use via WebAssembly?

This should be possible, but we haven't tried it. We're likely going to add a simplified target that has minimal dependencies (like no Eigen) that allows building on simple platforms.

Re: Developer preview of TensorFlow Lite

#76

Post by infnorm »

barbolo wrote: post #57
infnorm wrote: post #28

Earlier quoted context omitted.

The main TensorFlow interpreter provides a lot of functionality for larger machines like servers (e.g. Desktop GPU support and distributed support). Of course, TensorFlow lite does run on standard PCs and servers, so using it on non-mobile/small devices is possible. If you wanted to create a very small microservice, TensorFlow lite would likely work, and we’d love to hear about your experiences, if you try this.

Thanks for the answer. Currently I’m using AWS Lambda to deploy my TensorFlow models. But it’s pretty hard and hacky. I need to remove a considerable portion of the code base that is not needed for inference only routines. I do that so the code loads faster and to fit the deployment package size limit. If TensorFlow Lite is already a compressed code, then it may be much easier to deploy it to a serverless environment…

Sounds really interested. We're excited to hear about how that goes.

Re: Developer preview of TensorFlow Lite

#77

Post by infnorm »

d4l3k wrote: post #62
thepoet wrote: post #59

Is the Lite convertor also doing some sort of quantization or is it purely for file format conversion?

Tensorflow has supported quantization for a long time (and is recommended for mobile devices) so it very likely is.

Quantization comes in many different forms. TensorFlow lite provides optimized kernels for 8-bit uint quantization. This specific form of evaluation is not directly supported in TensorFlow right now (though it can train such a model). We will be releasing training scripts that show how to setup such models for evaluation.

Re: Developer preview of TensorFlow Lite

#79

Post by dangjc »

infnorm wrote: post #75
dangjc wrote: post #70

Earlier quoted context omitted.

Is it lite enough to compile with Emscripten and use via WebAssembly?

This should be possible, but we haven't tried it. We're likely going to add a simplified target that has minimal dependencies (like no Eigen) that allows building on simple platforms.

Cool. I have something else that uses Eigen in WebAssembly, so that hasn't caused any issues btw.

Re: Developer preview of TensorFlow Lite

#80

Post by rasmi »

barbolo wrote: post #58
rasmi wrote: post #33

Earlier quoted context omitted.

You can deploy TensorFlow model binaries as serverless APIs on Google Cloud ML Engine [1]. But I would also be interested in seeing a TensorFlow Lite implementation. [1] https://cloud.google.com/ml-engine/docs/deploying-models Disclaimer: I work for Google Cloud.

Thanks, @rasmi. I have a feedback for you guys. The pricing for predictions inference in GCP is not very fair. If I deploy a small model (like a SqueezeNet or Mobilenet) I pay almost the same price of someone deploying large models (like Resnet or VGG). That’s why I’m deploying my models on serverless environments and paying about 5 dollars for 1 million inferences. The pricing of GCP is: $0.10 per thousand predictio…

I see what you mean. To some companies, ML Engine's cost as a managed service may be worth it. To others, spinning up a VM with TensorFlow Serving on it is worth the cost savings. If you've taken other approaches to serving TensorFlow models to get around ML Engine's per-prediction cost, I'm curious to hear about them.
Post reply on HN