When your time is unscheduled it's not a problem. When you're scheduled to be in a meeting, or working with someone, it's disrespectful and potentially expensive.
If you never work with anyone else it's not a problem - all that will matter is getting stuff done. But when you work with other people it will be a problem. You will get the reputation of being unreliable, and a reputation once acquired is almost impossible to shift. Even devs end up interacting with other people. If you're any good, people will start coming to ask your opinion on things. If it's noticed that basically you're not there when you're expected to be there, you will get a bad reputation, even if you think that ass-in-seat time is irrelevant to getting things done.
And you may think it's unfair, but a reputation for unreliability in turning up on time will spill over into an expectation that you will be unreliable in other matters. That's how things work.
If you freelance or contract and fail to meet deadlines then you will lose clients, contracts, and money.
So it's entirely up to you. People under 30 have a much more relaxed attitude, but ask whether they have the wide range of experience necessary to make a nuanced decision about whether this is important. And will you only ever deal with people under 30?
I wouldn't have made you go home, but I would think twice about trusting you with anything important. If you can't show the discipline to be at work when you're asked to be there, how can I trust you with other aspects of self-discipline?
Added in edit: A lot of other commentators are saying "asses in seats is a dreadful policy" which is true, or "Provided you get your work done, who cares?", which is true. But once you are a member of a team, even if you don't formally interact with people in the code you're writing, you still have knowledge and experience that other people might want or need to tap into. If you're not there, you're depriving them of the ability to interact with you. Interactions in a team are not limited to formal meetings and formal pairings. In the best teams, it's the informal interactions that are the hardest for management to understand, the hardest to foster, and yet give the greatest rewards. If you're always late, you can break that.