Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Developer preview of TensorFlow Lite

developers.googleblog.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Developer preview of TensorFlow Lite

#51

Post by rajatmonga »

linuxkerneldev wrote: post #31
rajatmonga wrote: post #2

TensorFlow Lite is TensorFlow’s lightweight solution for mobile and embedded devices! TensorFlow has always run on many platforms, from racks of servers to tiny devices, but as the adoption of machine learning models has grown over the last few years, so has the need to deploy them on mobile and embedded devices. TensorFlow Lite enables low-latency inference of on-device machine learning models. Looking forward to yo…

> Looking forward to your feedback as you try it out. Thanks Rajat. We use typical Cortex-A9/A7 SoCs running plain Linux rather than Android. We would use it for inference. 1. Platform choice Why make TFL Android/iOS only? TF works on plain Linux. TFL even uses NDK and it would appear the inference part could work on plain Linux. 2. Performance I did not find any info on performance of TensorFlow Lite. Mainly interes…

1. The code is standard C/C++ with minimal dependencies so it should be buildable on even non-standard platforms. Linux is easy.

2. The interpreter is more optimized for being low overhead and the kernels are better optimized especially for ARM CPUs currently. While model performance varies by model - we have seen significant improvements on most models going from TensorFlow to TensorFlow Lite. We'll share benchmarks soon.

Re: Developer preview of TensorFlow Lite

#52

Post by sandGorgon »

rajatmonga wrote: post #2

TensorFlow Lite is TensorFlow’s lightweight solution for mobile and embedded devices! TensorFlow has always run on many platforms, from racks of servers to tiny devices, but as the adoption of machine learning models has grown over the last few years, so has the need to deploy them on mobile and embedded devices. TensorFlow Lite enables low-latency inference of on-device machine learning models. Looking forward to yo…

Hi, I have written about this before ( https://news.ycombinator.com/item?id=15595689 ) , but are there serialization fixes between cloud training and mobile ?

We have had huge issues in trying to figure out how to save models (freeze graph,etc) and load it on Android. If you look at my previous thread - it also mentions bugs,threads and support requests where people are consistently confused.

Re: Developer preview of TensorFlow Lite

#53

Post by pjmlp »

So it uses Bazel on Android....

Google devs, could you please get yourself together in one room and agree in ONE BUILD SYSTEM for Android?!?

Gradle stable, cmake, ndk-build, Gradle unstable plugin, GN, Bazel, ..., whatever someone else does with their 20%.

I keep collecting build systems just to build Gooogle stuff for Android.

Re: Developer preview of TensorFlow Lite

#54

Post by rajatmonga »

igorbark wrote: post #43
amelius wrote: post #37

I'm wondering if TF has something like pytorch's autograd. Does anyone know?

I only just briefly read the doc for autograd, but automatic differentiation is the strong default in TF if that's what you're asking.

Yes, it does have auto differentiation from day one. There's also a new autograd like functional API as part of eager. See https://research.googleblog.com/2017/10/eager-execution-impe...

Re: Developer preview of TensorFlow Lite

#56

Post by restricted_ptr »

petewarden wrote: post #34
lern_too_spel wrote: post #13

Earlier quoted context omitted.

What about using XLA to compile libraries for mobile deployment fusing only the operations needed by the model?

One nice thing about Lite is that it's a lot easier to just include the operations you need (compared to TensorFlow 'classic'), there's fusion for common patterns, and the base interpreter is only 70KB. That covers a lot of the advantages of using XLA for mobile apps. In return you have the ability to load models separately from the code, and the ops are hand-optimized for ARM. I'm still a fan of XLA, and I expect th…

How about quantization? Does tensorflow lite perform quantization or is it tensorflow supposed to do it? Is it iterative process or straightforward? Or are you training quantized models as nn api docs say?

Re: Developer preview of TensorFlow Lite

#57

Post by barbolo »

infnorm wrote: post #28
barbolo wrote: post #16

Would that be a viable option to deploy TensorFlow models on serverless environments (Lambda, Functions)?

The main TensorFlow interpreter provides a lot of functionality for larger machines like servers (e.g. Desktop GPU support and distributed support). Of course, TensorFlow lite does run on standard PCs and servers, so using it on non-mobile/small devices is possible. If you wanted to create a very small microservice, TensorFlow lite would likely work, and we’d love to hear about your experiences, if you try this.

Thanks for the answer. Currently I’m using AWS Lambda to deploy my TensorFlow models. But it’s pretty hard and hacky. I need to remove a considerable portion of the code base that is not needed for inference only routines. I do that so the code loads faster and to fit the deployment package size limit. If TensorFlow Lite is already a compressed code, then it may be much easier to deploy it to a serverless environment. I’ll be trying it in my next deployments.

Re: Developer preview of TensorFlow Lite

#58

Post by barbolo »

rasmi wrote: post #33
barbolo wrote: post #16

Would that be a viable option to deploy TensorFlow models on serverless environments (Lambda, Functions)?

You can deploy TensorFlow model binaries as serverless APIs on Google Cloud ML Engine [1]. But I would also be interested in seeing a TensorFlow Lite implementation. [1] https://cloud.google.com/ml-engine/docs/deploying-models Disclaimer: I work for Google Cloud.

Thanks, @rasmi. I have a feedback for you guys. The pricing for predictions inference in GCP is not very fair. If I deploy a small model (like a SqueezeNet or Mobilenet) I pay almost the same price of someone deploying large models (like Resnet or VGG). That’s why I’m deploying my models on serverless environments and paying about 5 dollars for 1 million inferences.

The pricing of GCP is: $0.10 per thousand predictions, plus $0.40 per hour. That’s more than 100 dollars for 1 million inferences.

Re: Developer preview of TensorFlow Lite

#60

Post by ianhowson »

infnorm wrote: post #29
mtgx wrote: post #17

How would this differ from uTensor? Did they make uTensor redundant? https://github.com/neil-tan/uTensor

We developed TensorFlow lite to be small enough to target really small devices that lack MMU’s like the ARM Cortex M MCU series, but we haven’t done the actual work to target those devices. That being said, we are excited when the ecosystem and community around machine learning expands.

Cortex-M compatibility was literally my first thought when I read this -- especially low-memory systems. Might have to hack it up myself.
Post reply on HN