A Google Doc ID (or Drive/Sheets/Slides) is a single identifier which denotes a resource in the cloud. It is a name; it is not intended as a secret. "Anyone with the Link" really means
anyone. In fact, I would say that "Anyone with the Link" access modes are a hack and a workaround for people who do not have Google accounts. If you rely on privacy, use Google accounts.
So if you created a secret, clandestine, world-readable, anyone-comments, blog post with the slug "http://example.com/correct-horse-battery-staple/" would you really complain when someone brute-force or outright guessed your blog post?
YouTube has "unlisted" videos which are unsearchable and unreachable unless you've been shared a link, but do people freak out when someone stumbles across this shared by others?
Just because a Google Workspaces ID is a long, jumbled-up, base-64 string doesn't mean it is a password or a cryptographically secure secret. It's quite different. It's meant to be well-known by anyone who is intended to have access. The "Anyone with the Link" access simply relies on a circle of friends/colleagues keeping the URL to themselves.