The article begins with "It’s a lengthy read, but don’t let the wall of text dissuade you." That's scary right there. Anyone who is scared off by this short, popular fluff piece needs a lot of help in reading (or is not a native speaker of English). To learn to read a piece like this thoughtfully and critically mostly takes practice in reading, not the latest "smart drug." I've been hearing people tout smart drugs for as long as I have been involved in online discussions (since 1992), and I have never seen any evidence of regular users of smart drugs doing anything smart.
Let's learn from how an indisputably smart person (LISP hacker and genuine NASA rocket scientist Peter Norvig, now Google's director of research) approaches claims of a new wonder drug. He writes of "Warning Signs in Experimental Design and Interpretation"
http://norvig.com/experiment-design.html
in a free online article that is also a wall of text, but which any Hacker News reader could read and apply to become more smart in day-by-day life. We can learn together by comparing the popular article kindly submitted here to the checklist of warning signs provided by Peter Norvig.
Warning Sign D1: Lack of a Randomized Controlled Trial
The article about "smart drugs" submitted here doesn't cite any scientific literature from any country and makes no reference to results from randomized, controlled trials. As another participant here has already pointed out in a smart comment,
http://news.ycombinator.com/item?id=4516111
the author of the article's main text is in the business of selling smart drugs. He is not a neutral, objective source of information on the subject. He doesn't seem willing to share his evidence base for his claims, perhaps because he has none.
Warning Sign D2: Lack of Double-Blind Studies
The author refers to people taking various drugs while being aware of what they are taking, which skews their recollection of their performance and very likely results in invalid recollections. Most SCIENTIFIC studies of mental performance interventions are careful to be "double blind" and to have neither the subject nor the experimenter aware of which subject received the studied intervention and which received a sham intervention. And such studies have careful tests of mental performance (working memory, IQ, or the like). The article here is mere advertising puffery by comparison.
Warning Sign D3: Too Few Subjects
The author advertising smart drugs doesn't describe any careful studies, so of course he doesn't mention the number of subjects in any studies. Most so-called studies of smart drugs have been far too small to have any statistical power.
Warning Sign D4: The Wrong Subjects
People voluntarily self-administering smart drugs are VERY likely not a representative sample of the general population. If some people self-report that they became smarter after using the drugs (do their friends and relatives agree?), that by no means shows that most people will be better off if they take the drugs. Random assignment to the treatment condition from among a representative sample is necessary to show that the drugs have any general or consistent benefit.
Warning Sign D5: The Wrong Questions
Most of the supposed tests of smart drugs appear to use mere user self-reports rather than standardized measures of mental performance that show up in the scientific literature on psychology. Simply put, no high-quality treatise on human intelligence
http://en.wikipedia.org/wiki/User:WeijiBaikeBianji/Intellige...
has ever reported that smart drugs are good for anything.
Warning Sign D6: The Wrong Statistics
Again, most statements about smart drugs found online have no statistics at all. In any event, what studies there have been about smart drugs that use the language of statistical analysis have mostly been very amateurish in their statistical approach.
Warning Sign D7: Lack of a Specific Hypothesis, or Overzealous Data Mining
Before the user begins taking a smart drug, is the expected effect "creativity," or "memory" or cognition, or what? The article, by its incoherence in grouping drugs by their supposed effects, shows that most of the time people who have sunk money into smart drugs are expected to imagine their own benefits, and will, to avoid embarrassment at being snookered.
Warning Sign D8: Lack of a Theory
The human brain is a very complicated biological system, with many feedback loops, and so far there is no mainstream theory of neurophysiology (a topic with MANY investigators all over the world) that suggests simple chemical interventions to produce long-term improvement in any aspect of mental performance.
Warning Sign D9: Lack of controls
It's not clear in the smart drug promotional literature even what should be controlled if there were a controlled trial of smart drugs. The smart drug promoters are not making enough effort to contrast taking smart drugs with other interventions (say, learning to read more challenging reading material by practice) that likely have better impact on long-term mental performance.
Warning Sign I1: Lacking Repeatability and Reproducibility
The author basically acknowledges that many smart drugs have results that are not reproducible reliably.
Warning Sign I2: Ignoring Publication Bias
Because smart drug trials are not announced in advance in publicly readable databases, every person selling smart drugs gets to announce unverifiable anecdotes while hiding in the file drawer all the cases in which smart drugs failed.
Warning Sign I3: Ignoring Other Sources of Bias
The smart drug promoters make no effort to subject smart drug claims to the statistical tests that in recent years have revealed many incorrect claims in published scientific literature on other subjects. Because the smart drug claim is an extraordinary claim, smart drug promoters should produce extraordinary evidence to back up the claim.
I'll skip Norvig's next warning sign, as the following one
Warning Sign I5: Taking p too Seriously
is more relevant for the claims here. There is NO indication that smart drugs have any real-world significance (in other words, that people actually live smarter lives after taking them) as contrasted with claimed (but unreported here) instances of the drugs having "statistically significant" effects.
The last few checklist items on Norvig's checklist
http://norvig.com/experiment-design.html
are also worth looking at in connection with claims about smart drugs.
To sum up, I am willing to do a mental test (memory, or IQ, as the smart drug promoter chooses) duel with anyone who claims that smart drugs have helped him become smarter. I don't believe it, because I don't think there is sufficient evidence to back up that extraordinary claim. If anyone wants to set up the duel, we should each choose "seconds" to negotiate the terms of the mental test duel. I would, of course, insist on a series of STANDARDIZED mental tests administered by a psychologist with a lot of clinical experience. I don't expect any user of smart drugs to win such a duel.
AFTER EDIT: Responding to the first two replies received, there are standard methods in statistics for estimating adequate sample sizes based on expected effect size of the intervention. There isn't any one best sample size that fits all experimental studies, but a careful experimenter will calculate what sample size is large enough to maximize likelihood of meaningful results (statistical power) while minimizing the expense of gathering lots of subjects for the experiment. My proposal for a duel is, of course, a sample size of 1 plus the number of smart drug users who come forth to take the duel, and has other problems as a data-gathering measure, but I offer the duel to illustrate that I think that anyone can improve in smarts, for any real-world purpose, in a lot of ways that are better than taking poorly understood drugs pushed by smart drug hucksters.