The refineries were there first, well before these areas were populated at all. I don't think that should give them a forever license to massively pollute. Back when they were built, and started to become populated the health risks were unknown... the people living there now mostly grew up there, and there was never a conscious choice to move somewhere dangerous.
In the case of Richmond, the poor black communities directly downstream of the plant are mostly descended from the workers brought in to build liberty ships during World War II. This was during a time when black people were legally restricted from getting decent jobs other than these temporary wartime exceptions, so moving to this place was their only opportunity to get a decent job. They lacked the legal freedom to make a choice about where to live in the sense you are implying, because of racist laws.
As a society, it is time to look at phasing these refineries out entirely, rather than relocating them. We already have the technology to do transportation in other ways, and it's already much cheaper when you count all of the health and environmental externalities. Already, 50% of diesel fuel in California is from renewable biomass, so you don't need an electric car for non-petroleum transportation. One of the north bay refineries was shut down a few years back, and converted to a renewable diesel plant.