Earlier quoted context omitted.
The problem is that this given permission doesn’t show in Files & Folders, and after turning it on and off there it still persists. The only way to revoke it is using some CLI command and restart the computer.
You “feed” it the document. Same way you select a picture on iOS. It is your deliberate decision and intent to open the document with that application. That is totally different from the application having permission to scan and view anything in for example the downloads folder
1) Grant access to a photo
2) Identify which photos you've granted access to
3) Revoke previously granted access
macOS's concession to give access to whole folders at a time is necessary for real software to work, but they haven't done a good job of items 2 and 3.