Earlier quoted context omitted.
How does storing the public key on the key bastion make it useless?
The public key is used by the webserver to encrypt credit card data. If it was stored in the key bastion, it wouldn't be available to the webserver.
1) super secure server which handles only the processing of credit cards, handed off to it through a carefully specified API
2) webserver which talks to that server
Obviously, the webserver needs the public key. There's no reason to delete the public key from the private server, but not really any reason not to.
Now, let's say that, under certain circumstances, system 1 itself needs to add encrypted records. Now you need the key there too. You could 1) generate a different key-pair and track which key touches which, but for records generated by system 1 you're generating a keypair that might as well be a symmetric key, which you've objected to; or 2) you could encrypt those records with a symmetric key, but now you've got additional code paths to audit and maintain; or 3 you could send off a request to an external server but that seems to add more complexity and vulnerability not less. Just using the existing public key to encrypt those records doesn't seem like a bad idea - it's a message to the server in the future from the server now, and messages to someone get encrypted with their public key.
Now, you've got a system where you have the public and private key sitting on the same server (and being used on the same server), and if someone does manage to get into that secure server (never impossible) the description I initially complained about would be equally applicable to this setup as to the setup Linode had.