I have NEM2 solar, so obviously a heat pump is a big win. But I'm convinced the heat pump I installed a couple years ago would save money even without the solar:
I pay $0.30-$0.40/KWh for electricity, and $0.08/KWh for natural gas ($2.35/therm) *.
My heat pump has a COP of 3.62 when the outdoor temperature is 47F: it uses 1KWh of electricity to move 3.62KWh of heat into my home. The old gas furnace was 80% efficient ("AFUE"): it used 1KWh of gas to dump 0.8KWh of heat into my home.
So, at 47F, as long as the ratio between the cost of electricity and natural gas is less than 4.53x (3.62 / 0.8), the heat pump saves me money. In my case, this means I save money when electricity costs less than $0.37/KWh, which it does almost all the time.
At $0.30/KWh-electricity, I effectively pay $0.083/KWh-heat with the heat pump, a 17% total heating cost savings over the $0.10/KWh-heat the old gas furnace cost.
Heat pumps do become less efficient as it gets colder outside: at an outdoor temperature of 17F, my heat pump COP is only 2.44, which would cost more than my old furnace ($0.123/KWh-heat vs $0.10/KWh-heat).
Extrapolating linearly between the 17F and 47F COPs from the manual (it only gives those two points; this isn't strictly correct, but close enough), the temperature below which my heat pump starts to cost me more money than my old gas furnace is roughly 30F (3.0 COP). In the decade I've lived in the bay area, I've never seen it get that cold, which is why this is such a perfect climate for heat pumps.
* These numbers are from August 2022