Consider that sometimes people do actually want to clean up all artifacts of an installation, and don't want to go digging around in a bunch of machine-generated directories trying to guess what to delete.
I think I agree that the default behavior should be to leave all that data on the user's system. But I also think that apps should have some kind of associated file manifest, which you can use to automatically clean up after the app, or at minimum get a definitive list that you can go through manually or with a script.
It turns out that Apple installer packages do include a manifest (cutely called a BOM) of package contents, but users don't typically keep package installers around after completing installation successfully, and it says nothing about files that may be written by the app itself while in use.
I don't see how you could possibly claim that the user library "ruined" anything. Most apps need to save some kind of settings and/or transient data files. Where else would they go?