This isn't accurate, either. Most of us are decidedly not facing the direction of travel. Specifically, we are evenly split in large parts of the boat, as they have bench chairs that face each other. Or you are on the side of the boat facing out.
The ferry is actually a very odd boat in this, btw. As I am not clear that starboard/port are fixed. The boat literally goes in both directions, such that I'm assuming I don't have to know which is the "front" of it irrespective of the direction it is currently traveling.
I get your point, that things can be difficult. And I appreciate having jargon that is specific for an attempt to make things clearer. I just find this as the sole explanation kind of unsatisfactory. Seeing the history of the terms makes a lot more sense.