Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…
It's actually pretty trivial to speed up, if you have multiple documents to parse, you can use multiprocessing.
from multiprocessing import Pool
def parse(html):
result = []
soup = BeautifulSoup(html, 'html.parser')
for p in soup.select('div > p'):
result.append(p.text)
return result
with Pool(processes=16) as pool:
for texts in pool.imap_unordered(parse, my_html_texts):
for text in texts:
print(text)