It is generally assumed to follow from the fact that the United States Customs Service was established back in 1789, and its duties, from the very beginning, involved dealing with contraband.
I poked around a bit, and while the first SCOTUS decision addressing this head on seems to be from 1977, it goes into more detail:
"That searches made at the border, pursuant to the longstanding right of the sovereign to protect itself by stopping and examining persons and property crossing into this country, are reasonable simply by virtue of the fact that they occur at the border, should, by now, require no extended demonstration. The Congress which proposed the Bill of Rights, including the Fourth Amendment, to the state legislatures on September 25, 1789, 1 Stat. 97, had, some two months prior to that proposal, enacted the first customs statute, Act of July 31, 1789, c. 5, 1 Stat. 29. Section 24 of this statute granted customs officials "full power and authority" to enter and search "any ship or vessel, in which they shall have reason to suspect any goods, wares or merchandise subject to duty shall be concealed . . . ." This acknowledgment of plenary customs power was differentiated from the more limited power to enter and search "any particular dwelling-house, store, building, or other place . . ." where a warrant upon "cause to suspect" was required. The historical importance of the enactment of this customs statute by the same Congress which proposed the Fourth Amendment is, we think, manifest. This Court so concluded almost a century ago. In Boyd v. United States, 116 U. S. 616, 623 (1886), this Court observed:
"The seizure of stolen goods is authorized by the common law; and the seizure of goods forfeited for a breach of the revenue laws, or concealed to avoid the duties payable on them, has been authorized by English statutes for at least two centuries past; and the like seizures have been authorized by our own revenue acts from the commencement of the government. The first statute passed by Congress to regulate the collection of duties, the act of July 31, 1789, 1 Stat. 29, 43, contains provisions to this effect. As this act was passed by the same Congress which proposed for adoption the original amendments to the Constitution, it is clear that the members of that body did not regard searches and seizures of this kind as `unreasonable,' and they are not embraced within the prohibition of the amendment."
(https://scholar.google.com/scholar_case?case=610713613239826...)