Earlier quoted context omitted.
What type of datasets?
Things with janky APIs or built on abandonware like imageboards, niche forums, dead comment communities, data dumps etc.
doing this professionally? Just curious..
Beautiful Soup
81–90 of 95 posts
Re: Beautiful Soup
#82Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…
It's actually pretty trivial to speed up, if you have multiple documents to parse, you can use multiprocessing. from multiprocessing import Pool def parse(html): result = [] soup = BeautifulSoup(html, 'html.parser') for p in soup.select('div > p'): result.append(p.text) return result with Pool(processes=16) as pool: for texts in pool.imap_unordered(parse, my_html_texts): for text in texts: print(text)
You might want to try a different parser as well, I tried a basic performance test comparison in another comment (https://gist.github.com/MercuryRising/4061368) and the html.parser was very slow compared to the lxml-xml or xml parsers for bs4
==== Total trials: 100000 =====
bs4 lxml total time: 110.9
bs4 html.parser total time: 87.6
bs4 lxml-xml total time: 0.5
bs4 xml total time: 0.5
bs4 html5lib total time: 103.6
pq total time: 8.7
lxml (cssselect) total time: 8.8
lxml (xpath) total time: 5.6
regex total time: 13.8 (doesn't find all p)Re: Beautiful Soup
#83Would suggest checking out pyquery. It uses JQuery-like syntax. It's been around a long time and in my opinion, it's way easier to use https://pypi.org/project/pyquery/.
Re: Beautiful Soup
#84The author also wrote a novel called Constellation Games which I enjoyed a lot https://constellation.crummy.com/
Also, free online, the very entertaining short story “Let us now praise awesome dinosaurs”: http://strangehorizons.com/fiction/let-us-now-praise-awesome...
Re: Beautiful Soup
#85Earlier quoted context omitted.
It's actually pretty trivial to speed up, if you have multiple documents to parse, you can use multiprocessing. from multiprocessing import Pool def parse(html): result = [] soup = BeautifulSoup(html, 'html.parser') for p in soup.select('div > p'): result.append(p.text) return result with Pool(processes=16) as pool: for texts in pool.imap_unordered(parse, my_html_texts): for text in texts: print(text)
You might want to try a different parser as well, I tried a basic performance test comparison in another comment ( https://gist.github.com/MercuryRising/4061368 ) and the html.parser was very slow compared to the lxml-xml or xml parsers for bs4 ==== Total trials: 100000 ===== bs4 lxml total time: 110.9 bs4 html.parser total time: 87.6 bs4 lxml-xml total time: 0.5 bs4 xml total time: 0.5 bs4 html5lib total time: 103.6…
If you want fast HTML parsing in python+lxml, use html5-parser https://github.com/kovidgoyal/html5-parser
Re: Beautiful Soup
#86Earlier quoted context omitted.
Things with janky APIs or built on abandonware like imageboards, niche forums, dead comment communities, data dumps etc.
doing this professionally? Just curious..
Yes, I'm a subject matter person who can code rather than a coder selling into a rewarding market. Someone with better business skills than I might find it more financially rewarding.
Re: Beautiful Soup
#87Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…
I work in finance and this thing is still indispensable for us. A LOT of financial info is only presented on a dynamically rendered HTML page that was last written in the bad old days.
Any poignant examples?
Re: Beautiful Soup
#88Earlier quoted context omitted.
On the off chance you were not aware, bs4 also supports[0] getting parse events from html5lib[1] which (as its name implies) is far more likely to parse the text the same way the browser would 0: https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.... 1: https://pypi.org/project/html5lib/
BeautifulSoup is an API for multiple parsers https://beautiful-soup-4.readthedocs.io/en/latest/#installin... : BeautifulSoup(markup, "html.parser") BeautifulSoup(markup, "lxml") BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml") BeautifulSoup(markup, "html5lib") Looks like lxml w/ xpath is still the fastest with Python 3.10.4 from "Pyquery, lxml, BeautifulSoup comparison" https://gist.github.com/MercuryRi…
You want a proper html 5 parser that can handle non valid documents. And the fastest one is https://github.com/kovidgoyal/html5-parser over 30x faster than html5lib
Re: Beautiful Soup
#89Re: Beautiful Soup
#90Earlier quoted context omitted.
doing this professionally? Just curious..
Yes, I'm a subject matter person who can code rather than a coder selling into a rewarding market. Someone with better business skills than I might find it more financially rewarding.
That sounds interesting.