GitHub's permission system is quite brittle here: Anyone with write access to a repository can silently swap out binaries on the releases page, which are then still listed as "Verified" if the commit is signed. It's a complex problem, but the current approach feels subpar.
Sounds like a good opportunity for integration with GPG, keybase, and other signing tools.
Of course current design leaves much to be desired.