I grabbed this from the first review article that popped up on google[1], so buyer beware. It says that non-repetitive sounds in general may create the effect:
Concerning tasks that do not involve auditory targets, studies with adults have consistently shown that especially short-term memory is sensitive to negative effects of noise. Immediate serial recall of visually presented verbal items is reliably impaired by task-irrelevant sounds (for reviews see Hughes and Jones, 2001; Beaman, 2005; Schlittmeier et al., 2012). Impairments occur with single talker speech and non-speech sounds such as tones or instrumental music, but not with continuous broadband noise or babble noise. This so-called irrelevant sound effect (ISE) occurs reliably even with low-intensity sounds, with meaningless speech (e.g., speech in a language unknown to participants), and when sound presentation is confined to a rehearsal phase after encoding of the list items. However, the ISE magnitude is determined by inherent properties of the irrelevant sound. Recall performance is specifically impaired by sounds with a changing-state characteristic, i.e., by auditory streams which consist of distinct auditory–perceptive objects that vary consecutively. For example, irrelevant sounds consisting of different syllables or tones evoke an ISE, whereas steady state sounds, e.g., continuous broadband noise or repetitions of single syllables or tones, have a minor or no effect.
In general, I've seen the argument that removing the effect can happen through habituating to repeated sounds, but also seem to recall some people showing that prolonged exposure to the irrelevant speech/stimuli before a task can also remove the effect--so maybe people wouldn't suffer as much with stimuli they're familiar with playing in the background. (there's also the issue of the things people actually do vs laboratory tasks).
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3757288/