Have you tried using object attributes/features to enable tracking instead of IOU-based tracking? I suspect this tracker falls apart when there are significant obstructions. Thoughts on a Recurrent YOLO or DeepSORT based tracker?
I've found that IOU based tracking is surprisingly reliable (my dataset was primarily surveillance footage with a static camera) if you take measures to smooth the data and predict object motion using a kalman filter.
Re: Show HN: OpenCV based multi-object-tracker with various object detection models
#21Kalman filter is also one good way to reliably keep track of objects under significant obstruction.