Medical consensus for most toxins/poisons is LNT - linear no threshold.
The basic idea is that you can perform some interventions at various doses, score the response, and then extrapolate to the origin. In most cases, simple linear regression gets you close to the origin, suggesting a linear response where even small doses has small harmful effects.
The trouble with this is that the data is rarely good enough to really support that model directly, instead you have to extrapolate the data to these smaller doses. This is simply due to variance in the data and errors in experimentation that make small effect sizes extremely hard to suss out. So you're always extrapolating from more extreme data.
For instance, a great deal of fuss has been made about alcohol, and whether moderate consumption (on the order of one drink a day) is actually beneficial. More broadly, this is related to the idea of hormesis, where small exposure to a poison actually confers a benefit, generally explained as coming from a compensatory response in the body. Every so often someone comes out with a new study or meta analysis that claims to be authoritative, but I still remain unconvinced one way or the other about alcohol.
________
The other issue here is the idea of relative risk. Even if you posit a non-zero harm from infrequent smoking, there are still many 'acceptable' risks that are likely far greater, such as that of road travel, air quality hazards, poor diet, lack of exercise, etc. These are all well understood and clearly outweigh the risk of infrequent smoking, but are 'business as usual', while a great deal of attention is paid to something like smoking, however infrequent.
In that sense, I would agree that it indistinguishable from background noise. But I would also say that there is likely real harm that results from this. In a broader decision-making sense, such minor harms do have a cumulative impact.