That would be an uncomfortable question to ask at the beginning of a long elevator ride, but it's no more uncomfortable than the following common and socially acceptable situation:
I'm at a fairly small club downtown. A woman is walking toward me, so I make eye contact and say "hi!".
She sits down next to me, and we begin to talk. She's very nice, but after a few minutes of conversation, I'm not attracted to her at all, and it's clear we have very little in common.
"Hacker789, you have a really cute smile. I'd love to get your number."
"Ah..."
I look down, and out of the corner of my eye, I see her nervously twisting her hair in her fingers.
I cringe just thinking about that, because it's happened to me before! But I would never shame her for asking me, and I would never charge her with making me feel "unsafe". That would make me into a villain.
And let's be clear—it can be dreadfully awkward to leave the table at a small club, often taking more than a minute to politely finish the task without hurting the other person's feelings too badly. It's even worse if your friend is at the table, happily talking to an acquaintance of the person you just rejected!
At least in an elevator, you're automatically free after the doors open.
Would I ever ask someone at the beginning of a long elevator ride a question that might make them uncomfortable? No, not if I could avoid it. In fact, I wouldn't ask anyone an uncomfortable question at the beginning of a long bus or train ride, either. But not every situation that has the potential to create discomfort is socially unacceptable; part of being a social creature is to take minor discomfort in stride, giving the other person the benefit of the doubt.
And the more frequently an uncomfortable situation happens to you, the better you get at handling it.