This is what I did and it works surprising well. I still have a decent camera and can use my banking app etc.
You could say that I actually have a pretty decent phone I just disciplined myself to use it as little as possible.
Edit: the part I am really missing is the compactness of a dumb phone. Years back I had a tiny Xperia Mini and liked the size of it. I would love to see a successor at around $300-350.
Please make it happen Sony