There are several competing philosophies of legal interpretation, but (afaik) only one, strict constructionism, wholly disregards intent as a principle.
Usually, the issue is more about whose intent should be considered, and how should that intent be ascertained, than whether or not "the letter of the law" is supreme.
The thing about it is that people are not governed by an ethereal intention that was in the legislature's mind when they crafted the law, because this is undefined and people may have differing opinions about what it is, and because there isn't necessarily a single unifying intention behind the passage of the law; there is only the law that was written, encapsulating the diverse intents of the various parties sufficiently to achieve passage. Assigning more than this enters the realm of speculation.
People do not vote their reps in or out based on their intentions, but on their work product (the laws that get passed and enforced). We do not want judges to be able to say "Yes, I know the law says this, but I think the law really meant this thing that is, in fact, very different."
That's way too much power for a judge, especially considering they're typically appointed, not elected (local judges are sometimes elected). The power of the people via the legislature can easily be neutralized by such actions, and confidence in the judiciary could be seriously harmed by a string of conflicting and contradictory rulings, making people unsure of what's actually legal/safe.
Thus, textualism considers the meaning of the law as written as a whole, and may consult documentation not related to the historical development of the specific legislation at issue, but related to the usage and understanding of words in context at the time that the law was enacted, to approximate what the law (not necessarily the legislature) really meant and how it would've been interpreted by the people who, through their representatives, approved its installation.
One could say this is all technical self-indulgence and that it misses the forest for the trees, and that would pretty well encapsulate the legal industry. ;)