> but I feel the "aggressor" is the party that first commits an unwarranted action
That right there is a problem at the heart of the concept of "aggressor": what time point do we pick as the baseline such that actions that lead to changes after that are aggression but actions before that are the status quo?
This played out in Europe with Alsace-Lorraine, where the Germans felt that the French took it away from them in 1919, while the French felt that the Germans took it away from them in 1871 and they just recovered it in 1919. Oh, but the Germans felt like the French had taken that territory away from them in the 17th and 18th century...
This is obviously a situation that is somewhat different from that of the Portuguese possessions in India. But it highlights the fact that "unwarranted" can be hard to define objectively. In practice, it tends to get defined by the victors writing the history books, though nowadays it can get decided by self-determination referenda and _then_ the victors (possibly of the referendum, possibly of resulting armed conflicts) writing history books.
From our point of view at the beginning of the 21st century, the Portuguese possession of Goa in the mid-20th was unwarranted but the British possession of the Falklands right now is... well, depends on whom you ask, but generally considered warranted. How much of that is because the population of the Falklands is happy being British subjects, and how much is because the British won the Falklands War? I wish I knew. :(
> If the cops are called on an individual for trespassing, would you call the cops the aggressor for trying to remove the individual?
No, I would not. But this isn't a perfect analogy either. I think it would be _very_ apt as an analogy if India had gotten a UN resolution that the Portuguese need to leave; even more so if UN security forces then threw the Portuguese out. Yes, I realize that this was not realistic at the time, for all sorts of reasons, not least of which was the US explicitly saying it would veto any such resolution, if I read the history correctly.
But as it was, the situation is more akin to the owner of the property being trespassed on seeing that the courts and the police won't help him out and personally kicking the trespasser off the property. Would that be described as "aggression"? It starts to depend on your biases, especially if the trespasser had been there for a while. This last is why the concept of https://en.wikipedia.org/wiki/Adverse_possession exists in English common law, so arguments over squatting and tresspassing won't continue ad infinitum. To some extent this gets applied in the international arena. Of course it doesn't always; then you get wars over territorial changes that happened hundreds or thousands of years ago; no shortage of those in recent history.
Anyway, at the time, there were a number of countries that condemned the Indian military action, including some which in principle supported the idea that the Portuguese possessions in India should become part of India. That is, they felt that the ends were good, but that the means were in fact military aggression and needed to be called out as such. We could try to fit that sort of dynamic into the individual trespasser being removed scenario; that would involve there being a dispute about whether the trespasser was in fact now more like a squatter, as well as dispute about whether forceful removal now, as opposed to peaceful removal, or at least removal by police, after some period of time gaining support for the removal was the better course of action...
Where all that leaves me personally is that I consider violent attacks on a previously broadly accepted status quo as "aggression" but I then have to allow that some "aggression" must be morally acceptable when the status quo either was unjust or has become unjust and no other solutions are available. Unfortunately, "unjust" is not an objective judgement either. :(
The situation with the Portuguese possessions falls into the "has become unjust" bucket for me, assuming the population of those areas wanted the Portuguese out. The Portuguese did not put this to a vote at the time, which is somewhat telling; I'm willing to assume based on time period and general zeitgeist that in fact the population did want them out.
Of course I would be happier to assume that sort of thing if I didn't know about counterexamples like the Falklands and Gibraltar. :( But as a point of comparison, Gibraltar _did_ have a vote on the matter in 1967, which is a pretty similar timeframe; presumably the Portuguese didn't put things to a vote because they were fairly certain they would lose.