For responding to user interactions (like filtering or searching) there isn't really a significant difference; like you said, it's just moving the "wait" elsewhere.
But many sites are more heavily read than interacted with: blogs, news, documentation, even to some extent HackerNews and comments. These are all "write rarely, read often" sites. In those cases, prerendering can be way faster both for the end-user (it's just HTML being downloaded from one single source) and also for the origin server (you build once, CDN caches it everywhere and takes over from there). Typically, the tradeoff is that it's also a PITA to manage these buildchains, especially once you get into obscure webpack or babel configs. Next.js handles it really elegantly.
Next.js has other benefits too, especially when coupled with Vercel. It is more than CRA + static builds, and even if you never end up using the rehydration system, the routing/image optimization/per-commit preview sandboxes may still be helpful, though not life-changing. For me the killer feature was being able to detach the data layer from the (write-rarely, read-often) frontend, such that the frontend could always just assume it would have access to the latest data from the API (because Next.js takes care of that).
To give you a before-and-after comparison... I worked on this page previously:
https://www.fieldmuseum.org/exhibitions
That version is running on Drupal. Some of it was in a Drupal template, some of it was in-house PHP. To fetch data, we had to use a mix of Drupal built-ins and some raw SQL, mixed into some ugly templating language. Then through custom modules we had to add jQuery and React, sprinkled on top. Drupal had to "build" the page into HTML whenever we save, and then a separate buildchain would add back the Javascript on top of that and try to bundle it all. The developer experience was ugly and required needed at least four languages (Drupal, PHP, jQuery, React). Filtering is done serverside, so filtering by e.g. Type = 3D movie requires an API call and takes several seconds. If you turn off Javascript the whole page breaks and you can't access any of the links anymore. This version is pretty fast thanks to in-house optimizations and extensive caching by Pantheon (a specialist PHP host), otherwise it would be really, really slow. Our dev and staging machines were hell to use because every page took like 10-20 seconds to load.
The new version (WIP, and I don't work there anymore): https://nextfield.vercel.app/exhibitions
That is 100% Next.js/React and only that, no more PHP or jQuery needed and no other frameworks. Data was moved to a headless CMS (DatoCMS in our case, which returned convenient GraphQL responses). It is slightly smaller over the network. All filtering is clientside and instant. If you go to a different exhibition page, it just has to load the JSON data for the new exhibition (text and image URLs + thumbnails), in like a 25kB JSON instead of the whole HTML page (headers and footers and all) all over again. The images are resized on the server for your viewport needs before they're sent to you (though TBH I am not a big fan of that feature because it's not preloading images right now). Even if you had JS disabled, the page would still load and the images and links would still work, you'd just lose the interactive filters.
So for users, the new version is hopefully a bit faster. The real improvement was in the developer experience, being able to code everything in React and not have to think about how it's going to get rendered into HTML or how we're going to balance our caching strategy (invalidations vs not overloading origin). And devs didn't need to use PHP at all anymore.
CRA wouldn't handle much of that, it'd just serve up a SPA run by a single server.