There was plenty of invention before patents existed as a legal concept.
Patents exist because some people who came up with clever ideas thought (properly) that it was in their best interest to be granted a legal monopoly on the implementation of their ideas and lobbied lawmakers to grant them these monopolies.
Patents did not come into being because we, as a society (speaking as an American), determined that our pace of innovation was too slow and decided that we needed to do something about it.
This idea, that patents encourage innovation, is the justification, not the motivation, for the legal construct that is the patent. There were no experiments done to test this hypothesis. In fact, I think the history of invention before patents suggests quite strongly that they are not needed; that, in fact, people will invent simply because we are tool builders and tinkerers.
EDIT: To testify to the fact that people invented before patents: http://bibliodyssey.blogspot.com/2011/12/machine-power.html
A book from 1606 showing illustrations of gear and hydraulic powered devices. Some are fanciful, others less so, but they all clearly show humankind figuring out every last way to apply a technology to the tasks of every day life.