Depends on what you mean by
defy.
What often gets lost in political debate is that there is Federal law enforcement and then there is State law enforcement. For the purposes of the US Constitution, local law enforcement is State law enforcement, even if a given State might draw a distinction themselves. It is helpful to remember that the US Constitution is architected to recognize three types of entities: US Government, States, and foreign governments. Off the top of my head I couldn’t tell you if it groups tribal governments into foreign governments or distinguishes them from those categories, but for the purposes of this conversation they’re closer to States than to foreign governments.
So whether you have counties, boroughs, townships, incorporated cities, school districts, water boards, parishes, independent cities, consolidated cities and counties, villages, capital territories or completely unorganized territory; none of this matters looking from the top down because those are all part of the “States”, and Federal law doesn’t really have to care about those details. As long as there is some kind of work organization there, the Feds can work with them, make agreements and negotiate with whoever has jurisdiction over whatever in whichever State.
What this also means is that States and/or their (for the purposes Federal convenience) subordinate entities can make laws and enforce them according to their own constitutions, laws and general policies, and the Federal government has its own parallel set of laws, courts, law enforcement officers, prisons and policies. Parallel, not higher. Your local police department is paid out of your local government’s budget, not out of the Federal budget.
So let’s take one of your example’s up there: marijuana. As far as the State of California or Washington is concerned, weed is fine. They aren’t going to spend money raiding pot farms, they’re going to tax them. According to their laws, it is generally legal for medicinal and recreational use, so you will not be arrested by a State law enforcement officer, dragged off to a State court and stuffed into a State prison for possession, use, sale or purchase provided proper procedures are observed.
The Feds though? They might still do all of that in this completely parallel legal system over here though. They can’t change* California or Washington law, but residents in these States are still subject to Federal law.
Well what about marriage? Well while Congress has the power to regulate substances offered for sale in this country, commerce clause and all that, the Constitution doesn’t say anything about giving Congress or the President license to determine what is and isn’t a lawful marriage in any of the States. According to the 10th amendment, this is clearly reserved for the States since it is a civil issue not specifically reserved for Congress. So unless you can twist the commerce clause to somehow cover marriage, then Congress doesn’t really have a say. No, this didn’t stop all 535 of them from sharing their opinions the very second Gavin Newsom issued that first same-sex marriage certificate, but that was about the extent of what they could do. All of the real debate was largely contained to internal State politics and about the most a Congressional candidate could give was their opinion.
What about the Defense of Marriage Act? Here’s where it got only slightly tricky, at least until it was determined to be unconstitutional anyway. Actually that’s pretty much the end of it: it was determined to be unconstitutional. While the States determine who can be married, Congress has given the Federal government the power to offer certain benefits including tax benefits to married couples, which is fine, and DOMA essentially barred same-sex married couples from receiving these benefits, and granted States license to ignore same-sex marriage certificates issued by other States which they would otherwise be obliged to recognize under some other clause of the Constitution.
Then SCOTUS struck down Section 3, later Section 5 and rendered Section 2 a casualty in the process. Basically the system worked as designed, even though it was law for close to twenty years.
It is fair to say, that when Federal law and State law are in conflict, Federal law does trump State law in most cases. What gets missed is that the Federal government must enforce Federal law. It can get States to help, by coercion or agreement, but sometimes that doesn’t work. That’s why Sanctuary City/State laws can also exist: the States are charged with enforcing their own laws without any expectation of assistance from the FBI or ICE, and it goes both ways: FBI and ICE have to be able to enforce immigration and other Federal laws, with or without assistance from any State they are operating within because something like immigration is explicitly under the jurisdiction of the Congress and Federal government. Sometimes there is a societal conflict, or values conflict, but unless there is a legal conflict, none of that actually matters to the Constitution and by extension, SCOTUS.
tl;dr: States/local govs employ their own law enforcement and enforce State/local laws. The Feds employ their own Federal law enforcement to employ Federal law. Neither actually works for the other, but rather in parallel and it is largely up to their discretion when and if to cooperate.
* The Feds sometimes make grants for very specific agreements, generally if they want a special kind if cooperation they have to pay for it, but otherwise your police compensation packages come out of your local budget.