Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"
Throughout human history, records have been forgotten, rewritten, changed, mutated, degraded, eroded away to nothingness. "The internet is forever" has always struck me as inhumane. Make a mistake or expose a weakness on the internet and it will always accompany you. It turns out that the internet is not always forever. I find that comforting.
"The Internet is forever" isn't some natural law by which all content abides as though it can never disappear. It's a warning that you don't control the content once it's accessible on the Internet.