Yes - I certainly didn't want to impart 'the obvious' in my response, but I'm not sure if there's an answer to your more literal question. If there was, we would have read an HN post on how there's a great new AI that develops compelling literary works, art, and music based on analyzing past masters and current psyche. Of course, there are AI writing, painting, and music composition systems, but other than seemingly random chance, they have yet to produce something compelling (novel, yes).
I do believe many great artists had a certain degree of mental affliction - not so much to incapacitate, but to provide additional insight. For example, it's been said that Ravel's Bolero was written in the early throes of his mental affliction, hence the repetitive theme. I completely agree with you - compositions (classical and modern) ultimately take us on small adventures from one place to another and back - happiness, despair, melancholy, and so on. As you mention, the melodies are the main 'characters' in the story, supported by chord sequences, rhythmic patterns, ostinato, and so on, all perversely using our ancient and deep auditory instincts of baby wails, hurt fellow tribe members, dangerous animals, thunder, laughter, etc. against us to elicit an emotional response. A great song creates a personal story in the listener's mind based on past experiences. Great painters and authors strive for the same response, but I think it's a more difficult task; for some reason we're more susceptible to auditory emotions vs. visual or the written word. Of course, soundtrack composers combine the visual with auditory very effectively; the whole being more than the sum of individual parts.
As for creating an entire world on which to base novels and characters on, Hemingway does touch on this as well, stating roughly that he could have made The Old Man and the Sea 1,000 pages with every detail exposed and examined (and that approach has successfully been used before), but it was much more compelling (and difficult) for him to eliminate all unnecessary things and have the town exist under the waterline as internal experience and just let the novel show whats important, which connects the reader with the story, as they have to fill in the details with their own experiences and ideas as opposed to Hemingway's.