More concerning would be a withdrawal from the ECHR, which focuses on fundamental human rights per se, not fundamental rights arising out of EU membership.Both are concerning, though. I was recently educated by a real lawyer about my idealistic view of the ECHR. It turns out that I had a fundamental misunderstanding for years, so I will share the essence of what they told me in case I wasn't the only one. (I am not a lawyer, so please forgive any continuing misunderstandings in what follows.)
The thing about the ECHR is that it doesn't really operate as a law, more a benchmark against which other laws are measured. A UK court will attempt to interpret UK law in a way that is compatible with the ECHR/HRA where possible. However, if a law cannot be so interpreted and is not compatible with the ECHR, the most a court can actually do is issue a declaration of incompatibility, which is the political equivalent of being told off and put in the naughty corner. The law itself still stands. Usually after a declaration of incompatibility the government does acquiesce and modify the law to be more compatible, but there have been some notable instances such as voting rights for prisoners where they have refused to do so. At that point, regardless of any theoretical protections afforded by the ECHR, the practicality is that is has no more power.
The legal situation seems to be quite different as far as the Charter and enforcement through the CJEU go, because that mechanism appears to have real legal teeth in its own right. This is the mechanism that will almost certainly disappear as part of Brexit, and so it is important for the protection of rights and liberties that something analogous be enshrined in primary UK legislation. The government of the day could still change it later, because in this country we don't have a constitution that guarantees these sorts of rights and sits at a higher level of law, but at least they would have to do so actively and under full parliamentary scrutiny like any other law.