The Copyright Office is not an authority in this context, it's just an opinion. They did not make any "finding". To a judge they may as well be any other
amicus curiae.
My opinion on the matter at hand is this: Artists who complain about GenAI use the hypothetical that you mentioned, where if you can accurately recreate a copyrighted work through specific model usage, then any distribution of the model is a copyright violation. That's why, according to the argument, fair use does not apply.
The real problem with that is that there's a mismatch between the fair use analysis and the actual use at issue. The complaining artists want the fair use inquiry to focus on the damage to the potential market to works in their particular style. That's where the harm is according to them. However, what they use to even get into that stage is the copyright infringement allegation that I described earlier: that the models contain their works on a fixed manner which can be derived without permission.
Not to mention the fact that this position means putting the malicious usage of the models for outright copyright infringement at the output level above the entire class of new works that can be created by its usage. It's effectively saying "because these models can technically be used in an infringing way, it infringes our copyright and any creative potential that these models could help with are insignificant in comparison to that simple fact. Of course, that's not the actual real problem, which is that they output completely new works that compete with our originals, even when they aren't derivatives of, nor substantially similar to, any individual copyrighted work".
Here's a very good article outlining my position in a more articulate way: https://andymasley.substack.com/p/a-defense-of-ai-art