No it isn't. Congress is only in session part of the time. Lawmaking - as opposed to politicking - currently requires a quorum and that legislators actually go on the record with both arguments and votes. It's where the rubber of governance meets the road and politicians can't pander to all and sundry while shrugging off criticism by saying 'they misspoke' and so on, but must instead commit to some sort of position (even if they obfuscate that by engaging in bullshit legislative tricks like poison-pill amendments). When Congress is in session it's doing (in theory at least) the Nation's business. When it's in recess the politicians can rest, schmooze their constituents, and so on - conducting their own political business rather than the nation's.
Now all this sounds very fine and idealist, but for lawmaking to be valid and have credibility then there needs to be some sort of formality to the process so that the citizenry can find out what their elected representatives are actually doing on their behalf, eg by consulting the Congressional record. Of course much of what happens in the legislative chamber is theatrical most deals are negotiated and brokered behind closed doors, but the lack of accountability that inheres in such arrangements is why there is a requirement for formalities in the first place.
So because there are formal rules on how laws are made, whether Congress is in session or not does matter. When it's only being kept in session by a technicality without any actual legislative work being performed, that says to everyone looking that the party engaging in such behavior regards rules as an obstruction to be got around rather than a standard for governing their own behavior, and that attitude is likely to promulgate itself among the population in various ways.
If you think recess appointments are a silly anachronism, amend the constitution. Unfortunately, I feel there's a particular faction in the country that actively aims at the undoing of the union in favor of a weak federation of states or several regional confederacies. That would not be so bad if the USA were its own planet, but as it's an imperial-scale power sharing the world with several rivals and many smaller national and corporate actors, a deconstruction of the federal system will create all sorts of power vacuums, and you know how that turns out.
