The Federalist papers literally argued against what became of Bill of Rights.
> The Federalist Papers (specifically Federalist No. 84) are notable for their opposition to what later became the United States Bill of Rights. The idea of adding a Bill of Rights to the Constitution was originally controversial because the Constitution, as written, did not specifically enumerate or protect the rights of the people, rather it listed the powers of the government and left all that remained to the states and the people. Alexander Hamilton, the author of Federalist No. 84, feared that such an enumeration, once written down explicitly, would later be interpreted as a list of the only rights that people had.[citation needed]
Yet we have a bill of rights!
Yeah you can absolutely use the papers to get context about what their authors were thinking. However, as I've shown with just one of many examples, you should not use the papers to claim that all the founding fathers think that way.
Even one of the other Federalists, Hamilton, disagreed with your interpretation.
> Alexander Hamilton and the Federalists took a broader view. Hamilton famously argued that the Clause authorized spending, so long as “the object, to which an appropriation of money is to be made, must be general, and not local; its operation extending in fact, or by possibility, throughout the Union, and not being confined to a particular spot.” While Hamilton did not advocate a completely unbounded interpretation of “general welfare,” under which Congress could spend money for virtually any object it considers beneficial, he and the Federalists did believe that the Clause authorized a wide range of spending for purposes that go beyond Congress’s other enumerated powers, so long as they were sufficiently “general.”
> The debate between the Madisonian and Hamiltonian views continued throughout much of the nineteenth and early twentieth centuries. As a general rule, Democratic presidents and members of Congress tended to adopt positions similar to Madison’s, or slightly broader. Thomas Jefferson, Madison, James Monroe, James Polk, James Buchanan, and Grover Cleveland all opposed bills authorizing spending on local infrastructure and disaster relief projects, citing constitutional objections. By contrast, the Federalists, the Whigs, and the Republicans tended to take a broader view of congressional power, closer to Hamilton’s position. Whig leader Henry Clay, for example, argued that the Clause authorized his proposal for a wide-ranging “American System” of canal and roadbuilding.
You're claim that this is what it meant back then is clearly wrong, since even back then people like Hamilton disagreed with Madison. That's why using Madison as your only guide for interpreting the constitution is, well, not really a great argument at all.