What I settled on is I have a computer with a lower-end nvidia gpu that lets me debug code and do basic stuff but is impractical for training any large model or with lots of data (although since the incremental cost is only the electricity, you can often still get far by training overnight or when you're doing something else). And then I use AWS more deliberately once I've figured out exactly what I want to run. Otherwise, you're reserving a cloud GPU for all the setup and debugging.
If getting a gpu is not practical, I'd try one of the K80s or other cheaper GPUs on AWS for all your debugging and then port to the more expensive instances only if you need them. Doing this as a hobbyist will be relatively inexpensive. I think a lot of the cost incurred as a hobbyist comes from reserving a gpu when you don't actually need it.