Earlier quoted context omitted.
Obviously your ancestors aren't British so there's something to the "where are you really from" question. I've been asked similar things even as a White American.
It has rather different connotations in Britain. As far as I understand it from Americans I know, quite a few people in the US have a fascination with where their, and others', ancestors came from, so it could be a fairly innocuous question. Pretty much any American I know would happily rattle off an answer of 'oh, I'm half Irish, a quarter French, and a quarter Spanish' or something similar. Here in the UK, even tho…
I'll ask "Where are you from?" as a way to understand someone's heritage only if it's obvious, from accent or other markers, that they are literally from another country.