Git-Annex is a cool piece of technology, but my impression is that it works best for single-user repositories. So for instance, as @nolist_policy described in a sibling comment, managing all your personal files, documents, music, etc. across many different devices. I tried using it for syncing large files in a collaborative repository, and the use of "magic" branches didn't seem to scale well.
YMMV, but it works for my org. We're an archival institution and have been using git-annex for more than a decade as the storage backend for a digital repository system designed for long-term, robust preservation. Admittedly, we only have 15-20 staff; but +30TB of data, ~750K files (binaries + metadata), across hundreds of collection repos.
Back (long time ago) when I was looking into this, there was no KISS, out-of-the-box way to manage the Git Annex operations a Git user would be allowed to perform. Gitolite (or whatever Git platform of choice) can address access control concerns for regular Git pushes, but there is no way to define policies on Git Annex operations (configuration, storage management).
Might not be super hard to create a Gitolite plugin to address these, but ultimately for my use-case it wasn’t worth the effort (I didn’t really need shared Git Annex repos). Do you tackle these concerns somehow? I guess if people don’t interact with your repositories via Git/SSH but only through some custom UI, you might deal with it there.