Coral's Edge TPU products are built specifically for this kind of thing: https://coral.ai/ Hands-on video (4 min): https://www.youtube.com/watch?v=-RpNI4ZrfIM
Interesting have RPi's lying around might get the USB Accelerator
Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
71–80 of 83 posts
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#72Earlier quoted context omitted.
Can confirm not fun
Could you elaborate on some of the problems you had overall?
I've unsuccessfully dabbled in gstreamer in the past. I was doing a project this weekend, and the comments on this thread motivated to give it another shot .. after a couple of hours (2-4ish?), I was able to get video off the Pi to my desktop (on the same LAN) but the performance was pretty bad. I didn't optimize much yet but let me summarize the key issues I experienced with gstreamer these last few hours:
1) Very little documentation; poorly explained pipelines. I tried to read what docs I could find but things quickly devolved into trying out random gstreamer pipelines posted in comments. People don't explain why they use one particular element over another. So it felt like whack-a-mole.
2) Installing gstreamer on the Pi was a breeze. I wanted to pull video off the connected camera and sent to VLC on my desktop. Sounded like something that would work out-of-the-box? Nope. Kept seeing lots of stackoverflow comments of people stabbing in the dark, getting errors (or have the thing just sit there and not work) with very little feedback on what was wrong.
3) I have very little indication of what is hardware and what is software accelerated in my pipeline. I have no idea where latency is coming into my pipeline.
Overall .. my modern expectation for software frameworks is "batteries included" .. it is totally reasonable for sophisticated software tools to be complex .. but gstreamer is just not designed that way. While I got it to work, I see massive latency (likely because my pipeline is inefficient) and degraded quality (no idea why).
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#73Earlier quoted context omitted.
Issue isn't ZeroMQ. The simple/inefficient way to do it is to capture frames one at a time, and send them via ZeroMQ. Video is pretty bandwidth intensive .. the only reason things like YouTube work as smooth as they do is that they use codecs such H264/265 (which are proprietary unfortuantely) and stream compress frames over the network. Now doing the codec in software burns a lot of CPU as this is very math intensiv…
Aren't IP cameras capable of H264 already encoded the video before output it to the network? H264 video stream has very good compression ratio and shouldn't consume too much bandwidth.
In the project I used zeromq, we were not using external IP cameras. My experience with IP cameras is still full of some seconds of latency .. I have no idea why.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#74Earlier quoted context omitted.
I think the difference with the jetson xavier is the tensor cores. The xavier is different from the pi (and even the jetson nano), like 100x different.
The Raspberry Pi doesn't have any "tensor cores" at all . According to Wikipedia, it actually does have a "Broadcom VideoCore IV" GPU, but I don't think this processor is ever used for deep learning. So if you did inference on the Pi then it would have to be on the CPU; inference is slower even on a meaty desktop CPU than on a GPU, never mind the low-powered CPU on the Pi. That is all academic, as the whole point of…
You know what, you're right and I'm wrong.
I went back and looked carefullier, and I must have read "on the edge", then "testing it locally" then "integrating tensorflow" and thought they moved it. But it doesn't actually do it on the edge at all. I think I need to learn to read.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#75Earlier quoted context omitted.
The Raspberry Pi doesn't have any "tensor cores" at all . According to Wikipedia, it actually does have a "Broadcom VideoCore IV" GPU, but I don't think this processor is ever used for deep learning. So if you did inference on the Pi then it would have to be on the CPU; inference is slower even on a meaty desktop CPU than on a GPU, never mind the low-powered CPU on the Pi. That is all academic, as the whole point of…
You know what, you're right and I'm wrong. I went back and looked carefullier, and I must have read "on the edge", then "testing it locally" then "integrating tensorflow" and thought they moved it. But it doesn't actually do it on the edge at all. I think I need to learn to read.
As I said in another comment here, I and lots of other commenters misread it that way too. I definitely find it funny they took "on the edge" to mean "anywhere on my local network", rather than just on the actual device capturing the data.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#76I did something similar, but because i had no requirement to playback audio "real time", i opted for a simpler solution.
I run a simple video capture from a Raspberry Pi Zero W running motion, meaning all motion events are captured, including leaves blowing in the wind. The captured files are stored on a NFS share per camera.
On the server i then monitor the parent directory for every camera for new files, and run my object detection there, which in turn generates push notifications with a screengrab if certain objects are detected. It also stores a bounding box annotated version of the file. Not really needed except for figuring out why you got an alert without any clear reason.
doing it this way however allows me to save a bit on each camera, and use dedicated hardware for object detection on the server. I currently use an Intel Neural Compute Stick 2 (https://software.intel.com/content/www/us/en/develop/hardwar...), and while it is far from dedicated GPU performance, it is equally far from dedicated GPU power consumption.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#77Earlier quoted context omitted.
The Orange Pi AI Stick Lite looks really interesting. Here's the link: https://www.aliexpress.com/item/32958159325.html and it says the PLAI training tools are (now?) free on request.
Yeah, that's promising, although I don't think there's much hope of support if it doesn't work as promised. And I have doubts about the software quality. As a small example: if you follow Gyrfalcon's installation instructions for the basic Plai Builder, it sets up a udev rule that makes every SCSI device world-writeable. I realized that by accident later. And of course everything is closed-source. Gyrfalcon's own sit…
This is sadly typical, and while I understand they don't want the support burden of hobbyists I would have thought the OrangePI would ship in interesting enough numbers for there to be some kind of support.
It looks like the OrangePi 4B includes ones of these chips on board?
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#78Earlier quoted context omitted.
Shameless plug, check out DOODS: https://github.com/snowzach/doods It's a simple REST/gRPC API for doing object detection with Tensorflow or Tensorflow Lite. It will run on a Raspberry Pi. It actually did support the EdgeTPU hardware accelerator to make the Pi pretty quick for certain models. They broke something so I need to fix EdgeTPU support but it's still usable on the Pi withe the mobilenet models or inception…
Few questions: 1. Did you build this for your own use cases? Interesting side project? 2. How do you feel about the need for base64 being a requirement on the endpoints? Isn't GRPC the wrong medium for this? Also, what do you see as the main limitations right now? The models?
1. I built it to integrate with Home Assistant and security systems. I was trying to use Tensorflow on a Raspberry Pi and the dependencies were a nightmare. Tensorflow in general is a nightmare to compile and run IMO. I got to thinking, what if I could make all the deps inside of a docker container. What if I could run it remotely. It was born out of that.
2. As for base64, I'm not sure of a better way to support sending raw image data over JSON (in REST mode) In some ways I think GRPC is a better medium than JSON (it supports either) as GRPC supports sending the RAW bytes. What leads you to believe GRPC isn't the right transport? Plus you can do it in a stream format if you want to do a lot of video.
The only limitations I can think of are that Tensorflow supports a myriad of CPU optimizations so providing a single container image that has all the right options is basically impossible. I created one that has what I think are some of the better options (AVX, SSE4.X) and then an image that basically should run on any 64 bit intel compatible CPU. To get optimized options you need to build the docker container yourself which can take the better part of a day on slower CPUs.
With that said, I also provide ARM32 and ARM64 containers that actually run semi-okay on Raspberry Pis and and other ARM SBCs. I can run the inception model on a Pi4 on a 1080p image in about 5 seconds which is pretty good IMO.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#79Earlier quoted context omitted.
I started with an Rpi by itself. Then I tried a Coral USB stick. I also tried the Intel Neural Compute Stick 2. The Coral USB accelerator doesn't accelerate all of the layers, only some of them. The CPU has to do the rest of the work. Plus, you only get this speed if you preload an image into memory and blast it through the accelerator in a loop. This ignores getting the image INTO the accelerator, which requires res…
> The Coral USB accelerator doesn't accelerate all of the layers, only some of them. My understanding is that with the pretrained models, everything happens on the TPU. If you use some lightweight transfer learning techniques to tweak the model [1], the last layer happens on the CPU. That's supposed to be insignificant, but I haven't actually tried it. I'm very curious what you're using for a model. You're clearly fu…
> My understanding is that with the pretrained models, everything happens on the TPU.
Nope. Try running SSDMN on a laptop with the stick and on a pi, you will get different scores due to some layers running on the host CPU.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#80Earlier quoted context omitted.
> Nowadays, with a lot of machine learning inference happening on the cloud Right, and if it's not on the cloud, it runs locally, as everything did before "cloud" became popular. We don't need to call it "edge" just to raise VC money or put out some PR. We can just say it runs locally, on-device, etc. If (big if) and when Adobe realizes that their Creative Cloud was a bad idea, are they going to call the next product…
> Right, and if it's not on the cloud, it runs locally, as everything did before "cloud" became popular. We don't need to call it "edge" just to raise VC money or put out some PR. We can just say it runs locally, on-device, etc. To me, "edge" means more than just "not cloud". It's appropriately used when making the point that computations happen where the data is gathered and the output is required (which seems actua…
> making the point that computations happen where the data is gathered and the output is required
This is how literally everything was done before the internet. It shouldn't be thought of as a new fancy concept.