Dirty bombs aren't an issue. You might want to ask yourself why we've never seen one used, especially if you understand how easy it is to obtain the necessary materials. IIRC we've only caught a handful of devices (none exploded). Also we have to consider when it explodes, it just doesn't make enough radiation. It is pretty difficult to generate enough radiation that would actually cause an increase in cancer rates (bombs disperse the radiation quickly, though it does aerosolize it). And then that cancer hits in 20+ years. It just isn't an effective weapon. Sure, it makes for a scary pipe bomb, but at the end of the day it is far more dangerous for the maker and greatly increases the complexity of the terrorist attack (also greatly increasing the likelihood that you get caught before you can deploy).
So a terrorist just gets a slight increase in fear factor, but it doesn't increase deadliness of the weapon, requires a lot more work, greatly increases the chance that they get caught, puts them in more danger, and so they just don't do it. There's too many downsides for only a minor upside. Why do it when you could just make a dozen pipe bombs and put them in trash cans around the city?
Really just think if you were a terrorist and wanted to do damage. There's a lot of things you could do (bunch of rusty nails on the I-5) that just don't happen. They are easy to accomplish, can do a lot of damage, but just don't. Why? Different objectives and just terrorism is extremely rare in the first place. And I'm pretty sure a drilling through feet of concrete and metal in a likely highly monitored site is too big of a hurdle when you can just go to home depot and get some stump remover and some steel pipes.