Why can't you have everything set up so that when you vote, you get what amounts to a JSON Web Token to be able to later verify that you did in fact vote? You could use the governments publicly available key to verify that your vote reached the central service, and part of the JWT could contain your vote as well as your identifying information (SSN in USA).
Obviously everything could have fancy UIs created for end users so they don't see that really all have is a JWT (maybe a QR code printed out when they vote? And all the info easily human readable?). Verification could be handled by a .gov address and also through manual use of the public key (so other services could be set up to verify votes as well). And internet connectivity wouldn't be a problem as they could just require T1 lines at polling locations (I assume if phones went out across the country the election would be delayed regardless). You could likely tell if someone had stolen the private key (the only way I can think of breaking this system), if you have a service to verify someone's vote, and it doesn't show up there, even though you have a signed JWT containing your vote. That would prove someone had stolen the private key, allowing for a makeup election.
Am I missing something basic of how this would be hackable? I'm one of those who finds it odd that many elections around the world are susceptible to simple human mistakes/purposeful malicious actions when it comes to counting ballots.