The Norse had a big fear of curses, the evil eye, the "strength in weakness". I think there's a wide theme in Norse legends, which is about spite and betrayal, but it doesn't work quite like we're used to. In Rigsthula, the social origin myth of the Norse, the first king is suggested to have taken the inheritance of his wealthier brothers by force - possibly by murdering them. And in the Norse creation myth, the gods also arguably seize the world from its original owner (and creates the world as we know it from his corpse).
So the theme is that all power is illegitimate, or at the very least seized/stolen, and the robbed want it back - and they
will get it back eventually. All hubris will fall, not just for the individual non-god as in Greek mythology, but for the whole world and the gods themselves. The world three has tree roots, one to the well of the norns (fates), one to the poisonous worm Nidhogg who gnaws on the root and will eventually kill it, and one to Mimir's well, the well of wisdom, where you can
maybe learn secret tricks of gods and rulers to postpone the inevitable.
So spite, or nid, dark power to break rather than to rule, is the ultimate danger to kings and rulers. To invite it by acts of cruelty, especially against the weak, is to bring ill luck upon yourself. Your followers, too, believe deeply in this, so they may abandon you if you seem to "draw in bad karma".
But those who are weak, and have nothing to lose, can dip into the power of spite and hate, and do things which would be unwise for a ruler to do, such as poisonings, betrayals, or vicious cruelty. They aren't evil for doing so, it's just the way the world works - if you run afoul of this, it was your own fault for inviting their hate.
Even demand for safety can be scornful, and "nid". Kings are supposed to trust in their own strength, and to some degree accept living with threats hanging over their heads. King Nidhad, in the story of Volund, listens to his wife's advice and hamstrings Volund. It's arguably self-defense since Volund certainly hates them, but it's still a scornful, cowardly act - which Nidhad and his family end up paying dearly for.
So yes, with respect to the Finnish and Sami neighbors, they would have feared them because of potential curses, but it wasn't because they were a magical people as such, it was simply were weak.
But Christianity complicated things. Odin, like the other pagan gods, is himself subject to the laws of fate and must be wise for his own sake, but the Christian God is almighty. You do not have to fear dark curses if He is with you. As a practical matter, they were a lot more willing to build walls and engage in other "cowardly" acts of self defense, and they could get away with it because their Christian followers didn't worry (much) that this would invite fate backlash. They were also a lot less afraid of things like public executions. It made possible much higher concentration of political power.
And no, the Norse didn't view that as simply fat idiots ripe for plunder. They admired all the great walls and splendor which concentrated political power had managed to build in Europe - things they had very little of at home. They did plunder, yes, but that was like a fox eating hens in a henhouse - he's still worried about the farmer.