You can run TensorFlow Lite on a Pi with no problems at all. You can even train and run basic gesture recognition with full TensorFlow on a Pi. Source: wrote a tutorial doing this for Arm ( https://github.com/ARM-software/ML-examples/blob/master/mult... )
There are gesture recognition sensors available that don't even need a general purpose CPU, eg. Broadcom APDS-9960. https://cdn.sparkfun.com/datasheets/Sensors/Proximity/apds99...