There's definitely debate over which specific scenarios tariffs could be beneficial, as I understand it, but the general theme is that any benefits are highly concentrated (one or two companies or industries will benefit) and the negatives are felt in a diffuse fashion(every consumer pays the tax). They are broadly protectionist and the ones that do exist usually are implemented for pretty specific reasons like the following:
1: A government wants to protect domestic industries over ones outside of the country by applying price increases to the foreign ones, with the idea being that the domestic industries just need to grow into being able to compete with the industries in other areas. This is called the infant industries argument. A central problem with this is that the industries will always benefit from the protectionist policy, and are unlikely to ever admit that they have "grown up" so to say. My general view on this is that groups will of course lobby to have benefits specific to their industry, and that there are probably scenarios where we would prefer to have things handled domestically rather than abroad, but I would generally want this to be highly targeted and time-gated(Once the industries are mature enough to compete, we wouldn't want to keep subsidizing them), and that other tools are probably more efficient for this purpose.
2: Some sort of national security argument, where production being cut off during war would be a serious concern. My general thought on this one is that if something is specifically important for national security, broad reaching taxes on all imports probably aren't as useful as targeted government interventions in those specific industries. The government can build whatever factories it wants or contract people to do specific things if it passes a law to do so. If we're worried that we need a domestic supply of beets(randomly selected example) and the government is willing to produce them at a loss for national security reasons, they should probably just do that rather than tax imports of coffee, chocolate, bananas, beets, beef, and cars in order to encourage domestic production of beets. The broad spectrum nature of across the board tariffs doesn't specifically protect any given industry, unless the specific protection desired is "nothing should be imported, only ever produced domestically."
3: Historically speaking tariffs were a major source of government revenue. There was no income tax in the very early US (and this was the case in many places), and tariffs were seen as an efficient way to raise a lot of money for the government. At the time it was also something that was a lot easier to measure than things like property value, sales, or individual income, because all the goods had to come in through a port. Pretty easy to check the majority of the things coming in, compared to other taxation methods. A major argument in the time period was actually that the government was making too much revenue, such that it was constricting the growth of the private economy! A huge debate in the 1880s and 1890s was on how the share of government revenue could be lowered, and the growth of the economy could be encouraged. Republicans argued that implementing more tariffs would actually reduce imports and lead to lower revenues, which was the stated goal of the McKinley tariffs.
The general reason some people oppose tariffs overall is that they represent an approach to economic growth based on zero-sum thinking, i.e. an idea that if another country experiences economic growth, ours must suffer economic decline. There tends to be more support from many people behind the idea that international trade allows multiple economies to grow in tandem, as I understand it, but I'm definitely not an expert in this stuff, haha. I just find the historical aspect interesting.
On your second point, describing it as a major export economy in the period I describe is maybe not capturing the scenario, because we were in the middle of a major change in manufacturing. We were major importers of manufactured goods in the preceding time period, and we exported agricultural goods! The period from 1890 to 1910 roughly(depending on when you draw the cutoff) is when the US mainly started exporting manufactured goods more than importing them, and it was a massive transition. The period we're talking about is probably best understood as when we were in the process of industrializing more.
It's fair to point out that the economy was pretty different at the time, but it was different in a bigger way.