Given that's the only thing Brexit actually changed, you should probably believe people when they say that's what they wanted (if they voted for it). You seem to be assuming that if you talked to some people who aren't articulate enough they didn't have real reasons?
The desire for this is reasonable. There are lots of cases where dumb decisions were taken outside the UK against the will of the local government, many of which led to long-lasting resentment:
1. Most obviously, the total refusal by the EU to let the UK control immigration, which basically crushed all wage growth out of existence for nearly two decades. Also see: Germany's brilliant decision to let a million migrants in (at which point they can freely move elsewhere). Also: inability to prioritize immigrants from other English speaking countries like Australia, Canada, USA.
2. Transcription of vague "human rights" principles into actual law, which led to a lot of notorious high profile cases where the government tried to deport e.g. radical Islamists and the EU courts rejected it, or where criminals ended up basically playing the government by making mendacious arguments under these laws. This was especially nasty because the UK and Poland saw this problem coming and got confirmation in the treaties that EU human rights law wouldn't apply to the UK, in writing, as clearly as possible, and then the ECJ overturned it anyway in a pretty flagrant bit of judicial activism.
3. EU cookie laws. 100% stupid, never supported by the UK government, now being rolled back.
4. Fishing quotas that allocated most UK fish to other countries. Hence why fish is such an electric issue post-leaving.
5. CAP, which is basically a way to force Brits to subsidize French farmers. See also: the extremely high level of EU controlled subsidies and spending that create large amounts of waste, fraud and general weakness in the business sector.
There are a whole lot of others people might bring up, that's just some examples.