I've learned to be okay with dicking around on my phone, but with some caveats:
* All notifications are turned off. No exceptions.
* Phone is only turned on when I'm home alone and not working or doing some focused activity. - Exception to this rule is that I can use my phone for its actual utility outside the house (e.g., phone calls, maps).
Rather than banning distraction altogether, I accept that it's inevitable and dedicate my phone to it. I find that this strikes a balance where I can spend quality time alone while still allowing for the occasional indulgence.