Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi • Hacker News Board

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#51

Post by dheera » Mon, Jun 22, 2020, 4:11 AM UTC

hikarudo wrote: post #38

dheera wrote: post #32

Can we all please stop using the term "edge" computing? It's nothing but a hype term and in reality it's really what we already had for the decades before the internet.

I disagree. The term "edge computing" actually adds precision to a description of a distributed system. Nowadays, with a lot of machine learning inference happening on the cloud, when seeing the term "edge inference" you immediately know you don't have to send heavy bandwidth-clogging video streams to the cloud. Inference on the edge is a clear trend in computer vision applications, now that we each year there are be…

> Nowadays, with a lot of machine learning inference happening on the cloud

Right, and if it's not on the cloud, it runs locally, as everything did before "cloud" became popular. We don't need to call it "edge" just to raise VC money or put out some PR. We can just say it runs locally, on-device, etc.

If (big if) and when Adobe realizes that their Creative Cloud was a bad idea, are they going to call the next product "Adobe Edge Edition! Wow you can actually run PhotoShop on your own desktop!"?

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#52

Post by nl » Mon, Jun 22, 2020, 4:16 AM UTC

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

Something seems weird here.

I agree that Python has some overheard, but the time taken should presumably be dominated by the neural network object detection. In TensorFlow that is written in (highly optimised) C, and should be using the NEON instructions on ARM[1].

Notably, DeepStream gives the same performance with the Python and C++[2].

YOLO inference speed is generally higher than a Mobilenet SSD, but you can run YOLO on TensorFlow instead of Darknet[3], or use a NNPACK version of Darknet.

Edit: "I don't need to turn on the noisy Xavier fan." - wait - this isn't on a Raspberry Pi? If you have a GPU on device then there's lots of other things going on.

[1] https://www.tensorflow.org/install/source_rpi

[2] https://developer.nvidia.com/deepstream-sdk (Scroll down for benchmarks)

[3] https://github.com/hunglc007/tensorflow-yolov4-tflite

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#53

Post by b34r » Mon, Jun 22, 2020, 4:43 AM UTC

What about package delivery people lol

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#54

Post by scottlamb » Mon, Jun 22, 2020, 4:54 AM UTC

nl wrote: post #49

scottlamb wrote: post #33

Earlier quoted context omitted.

> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" b…

The Orange Pi AI Stick Lite looks really interesting. Here's the link: https://www.aliexpress.com/item/32958159325.html and it says the PLAI training tools are (now?) free on request.

Yeah, that's promising, although I don't think there's much hope of support if it doesn't work as promised. And I have doubts about the software quality. As a small example: if you follow Gyrfalcon's installation instructions for the basic Plai Builder, it sets up a udev rule that makes every SCSI device world-writeable. I realized that by accident later. And of course everything is closed-source.

Gyrfalcon's own site is actively hostile to hobbyists. They only want to deal with researchers and folks preparing to package their chips into volume products. Signing up with a suitable email address and being manually approved lets you buy the device. You then have to negotiate to buy the Model Development Kits.

Hardware-wise, their stuff looks really neat. The $20 Orange Pi AI Stick Lite has the 2801 chip at 5.6 TOPS. Gyrfalcon's version of it costs $50. The 2803 chip does 16.8 TOPS. Gyrfalcon's USB-packaged version costs $70. That'd be a fantastic deal if the software situation were satisfactory, and a future Orange Pi version might be even cheaper.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#55

Post by scottlamb » Mon, Jun 22, 2020, 4:59 AM UTC

reichardt wrote: post #25

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

TensorFlow Lite with SSDLite-MobileNet gets you around 4 fps on a Raspberry Pi 4 (23 fps with a Coral USB Accelerator): https://github.com/EdjeElectronics/TensorFlow-Lite-Object-De...

You should be able to do a lot better than that if you're careful with the software. As I mentioned in another comment, the Coral USB Accelerator can do at least 100 fps. I haven't looked closely at that link, but likely they're doing H.264 decoding in software using one thread, then downsampling in software using one thread, then waiting for the Coral USB accelerator, and repeating. Maybe they also have the accelerator plugged into a USB 2.0 port rather than a USB 3.0 port.

The better approach is to use threading to keep all the Pi's cores busy and the USB accelerator busy at the same time, and to use hardware acceleration.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#56

Post by m463 » Mon, Jun 22, 2020, 5:43 AM UTC

quietbritishjim wrote: post #47

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

> You really want to use ... C/C++ for inference, not Python ... > You need ... C/C++, not Python. I think this is a red herring. Usually for deep learning you just use Python to plug together the libraries that actually do the processing, and those are written in terms of C/C++. You can see that in the article where the numpy array returned from OpenCV's video capture API is passed directly to tensorflow. Python nev…

I think the difference with the jetson xavier is the tensor cores. The xavier is different from the pi (and even the jetson nano), like 100x different.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#57

Post by scottlamb » Mon, Jun 22, 2020, 6:07 AM UTC

scottlamb wrote: post #33

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" b…

> * iirc, I got the Coral USB Accelerator to do about 180 fps with this model. [edit: but don't trust my memory—it could have been as low as 100 fps.]

Just dusted off my test program. 115.5 fps on my Intel NUC. I think that's the limit of this model on the Coral USB Accelerator, or very close to it.

My Raspberry Pi 4 is still compiling...I might update with that number in a bit. Likely the H.264 decoding will be the bottleneck, as I haven't set up hardware decoding.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#58

Post by staycoolboy » Mon, Jun 22, 2020, 6:15 AM UTC

scottlamb wrote: post #33

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" b…

I started with an Rpi by itself. Then I tried a Coral USB stick. I also tried the Intel Neural Compute Stick 2. The Coral USB accelerator doesn't accelerate all of the layers, only some of them. The CPU has to do the rest of the work. Plus, you only get this speed if you preload an image into memory and blast it through the accelerator in a loop. This ignores getting the image INTO the accelerator, which requires reshaping and shipping across USB. It fell to pieces with -one- 720P video stream. The NCS is worse.

I didn't bother with multiple $100 coral accelerators because why when I already have a Xavier?

As I said, my goal was 20-30fps with HD streams. Sure I could drop the quality, but I didn't want to, that was the point.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#59

Post by staycoolboy » Mon, Jun 22, 2020, 6:23 AM UTC

nl wrote: post #52

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

Something seems weird here. I agree that Python has some overheard, but the time taken should presumably be dominated by the neural network object detection. In TensorFlow that is written in (highly optimised) C, and should be using the NEON instructions on ARM[1]. Notably, DeepStream gives the same performance with the Python and C++[2]. YOLO inference speed is generally higher than a Mobilenet SSD, but you can run…

> wait - this isn't on a Raspberry Pi?

Literally the first sentence of my post.

You and many others seem to forget that I explicitly stated I wanted 4 HD streams at 30fps from my home IP address.

> but the time taken should presumably be dominated by the neural network object detection

There is a lot more to a pipeline than just inference.

The problem is aggregating the video streams, downsampling, submitting for inference, and then activating the Gstreamer element to write the MPEG. Most of this can use the nvidia memory properties on the nv* elements, which is great! However eventually you need to copy out for the Arm core. 4 HD streams is a lot of work for a small Arm core. The basic Gstreamer elements do not use the ACL AFAIK. I did recompile them with ORC optimization, but I'm not too familiar how/if that uses ACL/NEON. And the one that I built is basically just a bus messaging system for the downstream codec pipeline that is flagged by the tracker.

Can you point me to the benchmark in your link [#2] that indicates Python & C++ have the same performance? Nvidia does have an advantage in that Tensorflow-gpu supports them natively, but that is just for inference. I only see one table and the comments explicitly state they use the DS SDK and -not- TFlow.

Re: Tensorflow on edge, or – Building a “smart” security camera with a Raspberry Pi

#60

Post by staycoolboy » Mon, Jun 22, 2020, 6:40 AM UTC

thebruce87m wrote: post #27

staycoolboy wrote: post #18

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error…

I’d love to read a write-up of this.

Me too. :)

It is literally 90% in the NVIDIA SDK already. The demo examples provided with the kit read an HD stream, run inference and tracking -AND- provide an RTSP output! I started by replacing the HD stream with a videomux from my cameras into a composite image.

Working with the the Gstreamer RTSP server is hard, and NVIDIA basically hands it to you.

The next steps were to put tee elements on all 4 input streams to a selection demux and recompiled the tracker to send a bus event downstream to my element that controls the output of a video mux. This decides whether to send images to the final "videoconvert ! mp4mux ! payloder ! udpsink" pipeline that goes to a file... sort of.

It gets a little messy because I couldn't figure out how to start / stop gst's filesink so I send it to a UDP port instead and have another process on the machine grab payload packets and decide when to create a new file. It used to be one big MP4 file that was pushed to the cloud, and .. um, the program would crash and I would restart the process to get the next file ... i know ... currently I'm trying to chop it up based on idle time (e.g., no new frames in 300ms? start a new file!). It's ugly and I still get corrupt files sometimes, which is why I haven't written it up... and i'm lazy. I bet I could fix this if there was a manual to RTFM, but GStreamer is such a bear to work with and the only source of help is their weird mailing list archive.