Do you consume sugar, vitamins or oxygen? Have you ever inhaled Nitrogen Oxide or maybe stood near a car while its engine was on? Nevermind. This concern of long term health is reducible to: what effect does taking this action have on my probability of living for X more years at present functioning or better. Notice that merely being alive negatively impacts this, while when you are alive positively impacts the odds. Being sedentary and sitting for most of your day affects the probability very negatively. Is sitting most of your day worse for you than taking modafinil daily? I suspect being sedentary is far worse for you.
Or another, we can compare the effects of modafinil with getting into a car on a similar basis. With some basic knowledge of brain chemistry, claimed effects of the drug, understanding of placebo effect (~occurs in 1 in 3 or consider: drug OR Expectation/placebo Inferior to Drug AND Expectation), the drug's statistics, poor methodology in medical statistics and years of drug availability you can form a decent prior.
Based on my prior, taking modafinil is certainly not more lethal or less healthy than getting in a car and being near its fumes. So fear of health effects without any reasonable basis are not a rational argument to me. The supposed strong adverse effects? This is an argument for more understanding of how metabolisms and bodies differ, I have a confirmed strong allergy to nuts for example. Peanuts are much more dangerous to me than modafinil.
Yet I do not take modafinil. Not based on some unsubstantiated reverence for default biochemistry but instead on reasoned arguments. Its effects seem like they would impact creative thinking negatively. Furthermore, it is expensive and does not appear to have superior effects beyond being less addictive, a smoother delivery and a safer high dose cardiovascular profile than caffeine [1]. That is, modafinil does not appear to be any better at keeping you awake or focused than caffeine.
Similarly, I do not take caffeine because of its dependency forming nature and while it appears to boost working memory, enough studies indicate it antagonizes hippocampal function [2] to motivate me to minimize caffeine intake to every few weeks or less, as necessary. The mechanisms for modafinil are different so we can't expect all effects to be the same as caffeine. But it too seems to effect some improvement in working memory - most noticeable when the individual is lower performing [3] but according to [4] modafinil does show some negative effects on long term memory processes situated at the prefrontal cortex. in rats. Remember though, just because something appears in a medical journal doesn't make it true or accurate, especially when done on animal models or cell cultures. Nonetheless, the benefits in contrast with monetary costs and minute risks make modafinil not worth it and caffeine a very irregular thing for me.
Notice that I am not attacking your reluctance to take modafinil but rather, your opinion seems to be based on a nebulous definition of what a drug is and an assumption that the brain's architecture is optimal and any attempt to alter it will definitely be negative.
See also - Sulbutiamine. A vitamin B1 dimer.
[1] http://www.ncbi.nlm.nih.gov/pubmed/11862356
[2] http://www.ncbi.nlm.nih.gov/pubmed/19217915
[3] http://www.ncbi.nlm.nih.gov/pubmed/15221200
[4] http://www.ncbi.nlm.nih.gov/pubmed/20800665