I think that watching your "working" time decrease and being concerned about it is one sign that you're crossing into a new threshold of experience in your career. Most people in this position conflate "working" with "producing", but as many learn, these two are very different.
"Working", "producing", code grunt-ing, whatever you call it, seems to be what most young folks spend all their time doing in their first few professional years (it's what I did too). This is a good thing, it builds a lot of exposure and raw experience. You may be immersed in (or cranking out) code, fixing bugs, or working directly on your project/product for almost your entire work day, every day.
But at some point, many re-remember that they must learn and grow to advance in their field, and with that comes reading, exploring, trying/failing, as well as non-technical work, like planning/architecting, etc.
Once you start finding that you are pulled out of the code more and more, it's just a sign that "working" and "producing" are getting further from each other for you. And that's not a bad thing.