Well, here's a few points of comparison:
Suno's version of Mandate of Heaven [0]. This is my baseline, it was generated with their v4 model and so far it has remained my favorite AI generated song. I regularly listen to this one track and it brings me joy. There's many places where I think it could be drastically improved, but none of the competitors have managed to surpass it nor have they provided tools to improve upon it. The pronunciation is a bit bad sometimes and it fails to hold notes as long as I wish, but overall it has gotten the closest to my vision.
Eleven Music's version of Mandate of Heaven [1]. They don't allow free accounts to export or share the full song so you can only try a small fragment. It has much crisper instruments and vocals, but it has terrible pacing issues and pronunciation. The track is 4 minutes long, but the singer is just rushing through the track at wildly unexpected speeds. I cannot even play the song after it finished generating, so I haven't even been able to listen to the whole thing, it just gets stuck when I press play. Maybe some kind of release-day bug. The only tool that Eleven Music gives you for refining and modifying sections is "Edit Style", which feels pretty limiting. But I can't even try it because the track won't play.
Producer.ai's version of Mandate of Heaven [2][3]. This one has slightly worse instruments than Eleven Music, but the vocals are a bit better than Suno v4. It also has severe timing issues. I tried asking it to generate the track without a vibe reference [2] and also with a vibe reference [3]. Both versions have terrible pacing issues; somehow the one with the vibe reference is particularly egregious, like it's trying to follow the input vibe but getting confused.
It feels like AI song generation is just in a really awkward place, where you don't get enough customization capabilities to really refine tracks into the perfect direction that you're imagining. You can get something sorta generic that sounds vaguely reasonable, but once you want to take more control you hit a wall.
If one is willing to bite the bullet, there's a paid program for generating high quality synthetic voices while maintaining fine-grained controls: Synthesizer V Studio 2. But I haven't been able to try it out because I'm cheap and there's no Linux support.
The ideal workflow I'm imagining would probably allow me to generate a few song variations as a starting point, while integrating into a tool like Synthesizer V Studio 2 so I can refine and iterate on the details. This makes a lot of sense too, because that's basically how we are using AI tools for programming: for anything serious you're generating some code and iterating on it or making tweaks for your specific program. I would like to specify which parts of the track are actually important to me, and which ones can be filled with sausage in reaction to my changes.
Overall, Eleven Music generates instruments that sounds nice, but the singing leaves a lot to be desired (n=1). Eleven Labs is doing a ton of great product work so I'm really excited for the direction they'll take this once they're able to iterate on it a few times. A very strong showing for an initial release.
[0] https://suno.com/s/HfDUqRp0ca2gwwAx
[1] https://elevenlabs.io/music/songs/TGyOFpwJsHdS3MTiHFUP
[2] https://www.producer.ai/song/aa1f3cc4-f3e4-40ce-9832-47dc300...
[3] https://www.producer.ai/song/3d02dd17-69f1-41ba-a3ea-967902f...