late edit (2): added 3 notes including performance impact observation
I'm concerned about this behavior (both from privacy and performance perspectives), but I'm also not (quite) convinced this is working as described/implied here.
Before I get started: If you poke at this, open Console.app first. You can see recent logged "assessment" checks logged in "Mac Analytics Data" with the search "process:syspolicyd". You can use the same search to watch log messages (including all of the TLS negotiation etc.) for the checks in the device log.
The part that seems weird is that, if it is transmitting a hash (which seems possible/logical) the caching behavior doesn't appear to care or respect it?
The article suggests the following test:
echo $'#!/bin/sh\necho Hello' > /tmp/test.sh && chmod a+x /tmp/test.sh
time /tmp/test.sh && time /tmp/test.sh
I tried this test and got real runtimes of 0m0.289s and 0m0.006s. Then, I changed the file:
echo $'#!/bin/sh\necho Hellok' > /tmp/test.sh && chmod a+x /tmp/test.sh
When I re-ran the script, both runs are under 10ms. The content changed, but it didn't bother re-checking. I wrote the original script to a new file path:
echo $'#!/bin/sh\necho Hello' > /tmp/test2.sh && chmod a+x /tmp/test2.sh
This ran with runtimes similar to the original (0m0.232s and 0m0.006s). Same content, new path, new check. Here too, if it cares about the hash, it either isn't bothering to use it for caching decisions or the hash includes the path.
Then I tried rming the file, writing it again, and running it. Once again, it checks on the first request. I think this suggests it may be caching the result by inode? The author said they saw new checks after saving changes in TextEdit--I don't know much about TextEdit, but I'd guess it is doing atomic write/rename here.
Other random details I noticed:
1. it holds the connection open for a minute, presumably to minimize connection overhead for executions that'll generate many checks. My first checks were all in the 280-300ms range, but I tried one additional check within the minute and it only took 72ms. Making multiple requests in less than a minute may make it harder to notice
2. The device log has a "summary for task success" entry with pretty precise timing details on all parts of the request.
3. On my system, each of these attempts produces a "os_unix.c:43353: (2) open(/var/db/DetachedSignatures) - No such file or directory" error in the log from the libsqlite3 subsystem after the response comes back.
4. The "Mac Analytics Data" log entry for each request has a good summary that looks like:
assessment denied for test.sh
com.apple.message.domain: com.apple.security.assessment.outcome2
com.apple.message.signature2: bundle:UNBUNDLED
com.apple.message.signature3: test.sh
com.apple.message.signature5: UNKNOWN
com.apple.message.signature4: 1
com.apple.message.signature: denied:no usable signature
SenderMachUUID: ...snip...
5. When I add Terminal to the Developer Tools exemption on the privacy tab it does appear to kill the check. I'm not sure if there's genuine protection this check provides at some level, but I'll be considering adding either Terminal or at least some specific build tools to the exemptions I add on a new system.
6. After adding the Developer Tools exemption, if you have the app open, it'll ask if it can quit it for you. I took the hint and restarted Terminal. It'll do the same thing when you remove it from the list. But I didn't see the checks actually return until I rebooted. Also, my system froze during reboot. Hopefully a coincidence. :)
7. To put a better number on how this performance impact can compound for the kinds of builds I do all of the time, I ran `nix-build ci.nix` in the local directory for one of my projects before and after enabling the Developer Tools exemption for ~/.nix-profile/bin/nix. The run took 1m22s before, 45.5s after.
8. Looks like this is the same check as is run by `spctl --asses -v ` (at least, per the Console.app logs). That may make it easier to play with.