One should keep in mind, this is true of basicly any software whatsoever. It ALL infringes on patents, nearly without exception.
There is always legal risk in open sourcing code. Infact there is legal risk from pretty much any action whatsoever. Good, responsible companies don't let that become a barrier to doing the right thing.
If ARM really cared, GPU stuff would be open source. The fact it isn't pretty strong indication they don't. Don't just accept it when lawyers say no.
After all AMD and Intel both have full open source graphics stacks, and the world hasnt exploded yet.
Should also be noted, keeping stuff closed source is not very strong protection against reverse engineering, if it was DRM would actually work, and it never really has, dispite decades of effort. So its doubtful that keeping stuff closed is much protection against patent lawsuits.