I've chosen to do this twice now, and echo your sentiments
After saving for 2 years I quit my software job and spent two years driving from Alaksa to Argentina. I went on multi-day hiking trips, I poked lava with a stick, I paddled with icebergs, went scuba diving and camped in the mountains of Alaska and Patagonia.
I got up when I pleased, I cooked my own food (except when street food was stupid-cheap) and I worked on my passion projects (photography and writing at the time).
That experience changed my life, and after four more years saving money at a desk job, I quit again and spent three years driving around Africa.
I woke up to elephants and giraffes less than thirty yards from my Jeep-camper-house, I met tens of thousands of extremely friendly people and I ate delicious foods. I had adventures off the scale in the Congo, Sudan and Mali, I swam in white-sand paradise beaches in Mozambique and Ivory Coast and I again worked on my passion projects.
I certainly was "working" on the second trip - I wrote a couple of books, wrote for magazines, blogged, filmed video, etc. but I did it on my own terms. I could sleep in and take a rest day whenever I wanted, and I have not used an alarm for more than five years now.
Work on your own terms is so much more enjoyable, and life has a different focus. I highly, highly recommend people give it a shot!