Wait, if it is held in a separate dense array, is removal of a key from a dictionary O(N)?
Eventually if too many things are deleted you repack the array. Still amortized O(1). (No different than a hash table in general, which will need to recopy the underlying array when it grows.)