You're not crazy and not alone. At Hydraulic we do the exact same thing. It has other advantages you didn't mention:
• It yields much more collaborative code reviews, because for small tweaks it's quicker to just fix the thing you want changed than ask someone else to do it. After all, you've already got the code checked out and in your IDE. This is especially true when working with easily refactorable languages like Java or Kotlin and IDEs with great diff viewing support like IntelliJ, because if you don't like the symbol name someone picked you just change it. No need for a round-trip. This is huge because it stops reviewers/reviewees getting exhausted from "nitpicking" and fosters teamwork.
• You can see the IDE's static analysis outputs, which you won't see in a web based PR viewer. This can often let you spot bugs that might otherwise escape notice.
• You can easily run the unit tests, or add a quick test if you notice it's missing. If it fails, no problem, just commit the failing test, push and let the author know that they have more work to do.
To implement this workflow we use Gitolite on our "HQ" server that we use for everything. Everyone gets their own UNIX user and their own personal git clone of the product repos. Access controls are in force, so if you want someone to review something you push it into their repo but it has to be under a branch named rr/$PUSHER_USERNAME/change-name. So your PR queue is visible in your IDE's git branch explorer. Do a fetch and new work shows up to review, add commits or merge it and then delete the branch to close the review. The merge preserves the history of the review session.
We have it wired up to TeamCity and YouTrack (yup, I'm a JetBrains fan). Everyone also gets their own private TeamCity project. The combination of this means that devs can push private branches to get things nice and clean in CI without disturbing anyone else or polluting their UIs - there are no "draft PRs" or other self-contradictory concepts in this workflow. Ownership of branches is always crystal clear also, even for experimental branches or projects. Once a PR is merged into the tech lead's master branch, YouTrack notices the commit has gone green there and parses issue update commands out of the commit message, finally updating or closing the ticket automatically.
Note that this setup is cheap for small teams. TC and YouTrack are free for small setups, Gitolite is open source, all you need is a server. We pay 90 EUR a month for a dedicated machine at Hetzner that has 128 GB of RAM and 2TB NVMe flash, and everyone gets a UNIX user, so if you want to do remote dev on it or host random files you can do so and it's extremely fast.
Socially this arrangement is kernel-like. The tech lead of the project runs the repo from which releases are done and merge in from teammates, which in turn can merge in from their teammates and so on. Ownership is always clear because the act of merging is also the act of taking responsibility. You can't get a tragedy of the commons where juniors keep picking other juniors to review their work like you can in more conventional free-for-alls.