Earlier quoted context omitted.
This one is right. Have a shoe-box key, a key which is copied 2*N (redundancy) times and N copies are stored in 2 shoe-boxes. It can be on tape, or optical, or silicon, or paper. This key always stays offline. This is your rootiest of root keys in your products, and almost nothing is signed by it. The next key down which the shoe-box key signs (ideally, the only thing) is for all intents and purposes your acting "roo…
> It can be on tape, or optical, or silicon, or paper. You can pick up a hardware security module for a few thousand bucks. No excuse not to.
I'd rather one the most reliable and cheap hardware security model we know of: paper.
Print a bunch of QR/datamatrix codes with your key. Keep one in a fireproof safe in your house, and another one elsewhere.
Total cost: ~$0.1 (+ the multipurpose safe, if needed)