Is it really edge computing if the pi isn’t running tensorflow? I know the definition is kind of woolly. I wonder what the performance would be on a $100 jetson nano.
[deleted]
Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
31–40 of 83 posts
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#32Can we all please stop using the term "edge" computing? It's nothing but a hype term and in reality it's really what we already had for the decades before the internet.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#33I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video)
I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" because I haven't tested on this exact setup yet. My Macbook Pro and Intel NUC are a lot more pleasant to experiment on (much faster compilation times). A few notes:
* I'm currently just using the coral.ai prebuilt 300x300 MobileNet SSD v2 models. I haven't done much testing but can see it has notable false negatives and positives. It'd be wonderful to put together some shared training data [1] to use for transfer learning. I think then results could be much better. Anyone interested in starting something? I'd be happy to contribute!
* iirc, I got the Coral USB Accelerator to do about 180 fps with this model. [edit: but don't trust my memory—it could have been as low as 100 fps.] It's easy enough to run the detection at a lower frame rate than the input as well—do the H.264 decoding on every frame but only do inference at fixed pts intervals.
* You can also attach multiple Coral USB Accelerators to one system and make use of all of them.
* Decoding the 8 streams is likely possible on the Pi 4 depending on your resolution. I haven't messed with this yet, but I think it might even be possible in software, and the Pi has hardware H.264 decoding that I haven't tried to use yet.
* I use my cameras' 704x480 "sub" streams for motion detection and downsample that full image to the model's expected 300x300 input. Apparently some people do things like multiple inference against tiles of the image or running a second round of inference against a zoomed-in object detection region to improve confidence. That obviously increases the demand on both the CPU and TPU.
* The Orange Pi AI Stick Lite is crazy cheap ($20) and supposedly comparable to the Coral USB Accelerator in speed. At that price if it works buying one per camera doesn't sound too crazy. But I'm not sure if drivers/toolchain support are any good. I have a PLAI Plug (basically the same thing but sold by the manufacturer). The PyTorch-based image classification on a prebuilt model works fine. I don't have the software to build models or do object detection so it's basically useless right now. They want to charge an unknown price for the missing software, but I think Orange Pi's rebrand might include it with the device?
[1] https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc...
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#34This is extraordinarily neat.
Home Assistant does have a tensorflow integration [1] that allows you to run other home assistant automations (including various alerts, alarms, and scare sequences) based on person detection with basically any camera (since it's kind of a hub-and-spoke model to all other possible IoT devices).
[1] https://www.home-assistant.io/integrations/tensorflow
I struggled recently to get it running on my actual GPU since I run Home Assistant on a home server. I ended up making a custom component using pytorch instead on Pop OS 20.04 and it works gloriously. CPU usage way down and GPU has something to do now.
My super awesome self-hosted alarm system is now extra-super awesome.
Of course burglars are going to all just start wearing AI adversarial t-shirts.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#35I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…
Similar to other's comments, I'd love to read a write up about this.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#36I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" b…
>* I use my cameras' 704x480 "sub" streams for motion detection and downsample..
i've encountered cheap IPTV cameras where the main high-res stream was actually being offered with a time-shift compared to the sub-stream.
weird shit happens when you have a camera that does that, then you act on data from the sub-stream to work with data on the main stream. I played with a 'Chinesium' cctv with generic firmware that had such a bad offset that I could actually use a static offset to remediate it.
I assumed it was just a firmware bug, since the offsets didn't seem to move around as if it was a decode/encode lag or anything of that sort.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#37I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…
I think I was able to run Yolo 3 on my shitty $99 smartphone a while ago. Did it for human detection. Don't remember the FPS, but it wasn't 0.1, it was much better than that.
The beauty of a smartphone is it's all in one small package, and it has everything - the CPU/GPU + camera + 4G/3G + wifi, plus you can trivially hook it up to a huge USB powerbank. They even have weatherproofed ones.
RPi will cost you more with all the bells and whistles to actually make it work for this case.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#38Can we all please stop using the term "edge" computing? It's nothing but a hype term and in reality it's really what we already had for the decades before the internet.
I disagree. The term "edge computing" actually adds precision to a description of a distributed system. Nowadays, with a lot of machine learning inference happening on the cloud, when seeing the term "edge inference" you immediately know you don't have to send heavy bandwidth-clogging video streams to the cloud.
Inference on the edge is a clear trend in computer vision applications, now that we each year there are better low-power neural network accelerators.
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#39Earlier quoted context omitted.
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" b…
>* I use my cameras' 704x480 "sub" streams for motion detection and downsample.. i've encountered cheap IPTV cameras where the main high-res stream was actually being offered with a time-shift compared to the sub-stream. weird shit happens when you have a camera that does that, then you act on data from the sub-stream to work with data on the main stream. I played with a 'Chinesium' cctv with generic firmware that ha…
Yeah, that sucks.
Did the camera send SEI Picture Timing messages? RTCP Sender Reports with NTP timestamps? Either could potentially help matters if they're trustworthy.
I haven't encountered that exact problem (large fixed offset between the streams), but I agree in general these cameras' time support is poor and synchronizing streams (either between main/sub of a single camera or across cameras) is a pain point. Here's what my software is doing today:
https://github.com/scottlamb/moonfire-nvr/blob/master/design...
Any of several changes to the camera would improve matters a lot:
* using temporal/spatial/quality SVC (Scalable Video Coding) so you can get everything you need from a single video stream
* exposing timestamps relative to the camera's uptime (CLOCK_MONOTONIC) somehow (not sure where you'd cram this into a RTSP session) along with some random boot id
* allow fetching both the main and sub video streams in a single RTSP session
* reliably slewing the clock like a "real" NTP client rather than stepping with SNTP
but I'm not exactly in a position to make suggestions that the camera manufacturers jump to implement...
Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi
#40I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…
I think I was able to run Yolo 3 on my shitty $99 smartphone a while ago. Did it for human detection. Don't remember the FPS, but it wasn't 0.1, it was much better than that. The beauty of a smartphone is it's all in one small package, and it has everything - the CPU/GPU + camera + 4G/3G + wifi, plus you can trivially hook it up to a huge USB powerbank. They even have weatherproofed ones. RPi will cost you more with…
How did you set it up on the software side though? How flexible/customizable was it?