Transmission occurs when one has fully internalised a practice that one has learned from a master (of that practice).
You may be able to teach yourself meditation from a book, without a teacher. But that can't confer on you the confidence that you're doing it right. Many practices depend critically on confidence. Of course, you have to be confident that your master has really mastered the particular practice; you gain that confidence by knowing that the master received transmission, from someone who received transmission... etc. That line of transmission is referred to as the "lineage" of that practice.
There are no ritual formalities for mindfulness; every monk learns mindfulness, from other monks. The first monks learned it from the Buddha; but if you're going to learn the practice from a layman, you need to start inquiring into your teacher's transmission lineage.
In tantra, there are rituals that accompany the process of mastery: permission to study the text, a reading of the text, a commentary on the text from the master.
But these rituals are not the transmission.
There's usually some kind of "inner" transmission that depends on a personal relationship between the master and the student. But even that doesn't seal the transmission; the student still has to internalise the practice, which might take years.
It's very interesting to look into the process by which tulkus "produce" innovative practices. Often, the teaching is delivered to the tulku by enlightened beings, who learned the practice from some Buddha other than The Sage of the Munis. The tulku must then practice the teaching, usually for many years, until the insight stabilises, before it's safe for him to disseminate it.
Tulkus often have hard lives.