This is like saying it’s no big deal for everyone to carry around a machine gun because humans have been stabbing each other for centuries. Scale and lowered skill barriers are transformative: you’d have to have a pretty epic middle school grudge to spend hundreds of hours learning Photoshop well enough to produce equally realistic fakes, whereas there are now apps which would allow anyone with an iPhone to spend a few minutes getting the same result.
That matters because once you realize the false dichotomy you can discuss intermediate steps: for example, if businesses have legal requirements to watermark output, answer legal requests to identify the user who produced an image, or apply a robust NSFW filter, etc. that certainly wouldn’t prevent all abuse but it would make a difference for the majority of school kids who a don’t have the skills to setup an entire open source ML toolchain.
Where this especially matters is curated experiences. A lot of businesses would like to offer ways to personalize their services but they’re worried about what happens if, say, you can get Disney’s Make Your Own Magical Kingdom app show someone else’s kid your clever prompt. It’s not hard to imagine a regulatory framework similar to what’s evolving with self-driving cars where companies would need to validate their products’ performance at certain levels to be certified as suitable for certain thresholds (safe to show to the user, their friends, other people in the same game as them, fully public, etc.).
A big area will be brand safety: even if it’s not technically illegal or even unprecedented levels of violence to show a bloodied Mario and Luigi fighting, there’d be outraged parents and an epic lawsuit if your app’s generative AI feature can produce it, and that means there’s a big market demand for services which can assure buyers that they don’t have to worry about it.