Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beautiful Soup

crummy.com

Post reply on HN
Print view View original

61–70 of 95 posts

Re: Beautiful Soup

#61

Post by Der_Einzige »

djtriptych wrote: post #8

Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…

I remember that I needed to do something involving performance with beautiful soup. Switching the HTML parser backend (as they mention in the docs in BS4) gave me an order of magnitude speedup...

Re: Beautiful Soup

#63

Post by rnx »

I fondly remember being introduced to this library as part of the first project I worked on at my first development job. I was lucky there it was the right challenge at the right time.

Re: Beautiful Soup

#64

Post by kristianp »

mdaniel wrote: post #32
cpach wrote: post #27

Does anyone know if there as a good equivalent for Go?

I've heard https://github.com/gocolly/colly#readme mentioned fondly, but I've never used it

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
  e.Request.Visit(e.Attr("href"))
 })
Sometimes I wish Go idioms included an iterator abstraction, it's easier to understand and less hideous than that functional callback style.

Re: Beautiful Soup

#65

Post by js2 »

I used BS to scrape rogerebert.com and post his reviews to letterboxd:

https://letterboxd.com/re2/

I copied over only the first two paragraphs of each of his reviews with a link back to the original.

The HTML is a total mess having obviously moved around the web a couple times. So it required a bunch of cleanup. But that wasn't even the hard part. The hard part was getting the correct TMDB ID for the movies because his reviews also have either no useful metadata or metadata that's wrong, like incorrect movie years, misspelled actor names, etc.

I never was able to get API access to letterboxd, but they have a CSV import feature which worked-out well enough.

Re: Beautiful Soup

#67

Post by ravishi »

js2 wrote: post #65

I used BS to scrape rogerebert.com and post his reviews to letterboxd: https://letterboxd.com/re2/ I copied over only the first two paragraphs of each of his reviews with a link back to the original. The HTML is a total mess having obviously moved around the web a couple times. So it required a bunch of cleanup. But that wasn't even the hard part. The hard part was getting the correct TMDB ID for the movies because h…

I had my share of gigs where we just decided to scrape the old site with BS and extract structured data from there to render a new site. It was sometimes cheaper than dealing with their ancient ad-hoc cms monstrosities.

Re: Beautiful Soup

#68

Post by driscoll42 »

westurner wrote: post #49
mdaniel wrote: post #31

Earlier quoted context omitted.

On the off chance you were not aware, bs4 also supports[0] getting parse events from html5lib[1] which (as its name implies) is far more likely to parse the text the same way the browser would 0: https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.... 1: https://pypi.org/project/html5lib/

BeautifulSoup is an API for multiple parsers https://beautiful-soup-4.readthedocs.io/en/latest/#installin... : BeautifulSoup(markup, "html.parser") BeautifulSoup(markup, "lxml") BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml") BeautifulSoup(markup, "html5lib") Looks like lxml w/ xpath is still the fastest with Python 3.10.4 from "Pyquery, lxml, BeautifulSoup comparison" https://gist.github.com/MercuryRi…

I was curious, so I tried that performance test you linked to on my machine with the various parsers:

    ==== Total trials: 100000 =====
    bs4 lxml total time: 110.9
    bs4 html.parser total time: 87.6
    bs4 lxml-xml total time: 0.5
    bs4 xml total time: 0.5
    bs4 html5lib total time: 103.6
    pq total time: 8.7
    lxml (cssselect) total time: 8.8
    lxml (xpath) total time: 5.6
    regex total time: 13.8 (doesn't find all p)
bs4 is damn fast with the lxml-xml or xml parsers

Re: Beautiful Soup

#69

Post by wildrhythms »

diarrhea wrote: post #23
pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

For JS, I've used Selenium with a Chrome driver, then parsed the HTML with BeautifulSoup. I know nothing about web development, so this might be an outdated way, but it worked. BS was nice to deal with.

I used Puppeteer for this to great success. Very easy to set up, and you control the whole browser + access to everything you would have access to normally through Chrome dev tools.

Re: Beautiful Soup

#70

Post by js2 »

ravishi wrote: post #67
js2 wrote: post #65

I used BS to scrape rogerebert.com and post his reviews to letterboxd: https://letterboxd.com/re2/ I copied over only the first two paragraphs of each of his reviews with a link back to the original. The HTML is a total mess having obviously moved around the web a couple times. So it required a bunch of cleanup. But that wasn't even the hard part. The hard part was getting the correct TMDB ID for the movies because h…

I had my share of gigs where we just decided to scrape the old site with BS and extract structured data from there to render a new site. It was sometimes cheaper than dealing with their ancient ad-hoc cms monstrosities.

After I managed to wrangle the review text from the HTML it still needed this sort of cleanup:

    def clean_text(text):
        text = re.sub(r"[\x7f-\x9f]", "", text)  # remove control chars
        text = re.sub(r"[\xa0\r\t]+", " ", text)  # replace with spaces
        text = re.sub(r"\n+", "\n", text)  # squash runs of newlines
        text = re.sub(r"\s+", " ", text)  # squash runs of spaces
        # Remove newlines unless they appear to be at the end of a sentence
        # or if the sentence is shorter than 80 characters.
        text = re.sub(r"([^.?!\"\)])\n", r"\1 ", text)
        text = re.sub(r"\n([^\n]{,80})\n", r"\1 ", text)
        return text.strip()
Post reply on HN