Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beautiful Soup

crummy.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Beautiful Soup

#21

Post by phone8675309 »

werds wrote: post #11
phone8675309 wrote: post #7

How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)

its not useful in those cases, but usually for those js rendered sites you can replicate the ajax requests which happen and get nicely formed json documents to parse through instead.

I used to do that when I was responsible for a set of web crawlers to extract public records data, but the problem is that changes happen and these sorts of things become out of date fairly quickly.

Getting this working in a headless browser driven by Selenium would probably be easier for maintainability.

Re: Beautiful Soup

#23

Post by diarrhea »

pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

For JS, I've used Selenium with a Chrome driver, then parsed the HTML with BeautifulSoup. I know nothing about web development, so this might be an outdated way, but it worked. BS was nice to deal with.

Re: Beautiful Soup

#24

Post by driscoll42 »

pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

If you don't need JS, I think it's still the best. Sure, that doesn't always work for you, but once you start needing to render JS the scraping slows down tremendously.

Re: Beautiful Soup

#26

Post by sergiotapia »

phone8675309 wrote: post #7

How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)

If you're lucky those sites have the raw data a server side generated JSON payload right in the site source code markup.

For example Target is clientside, but has all the data in a `window.FOOBAR = json` variable you can fetch and parse with some substring magic. Much easier than spinning up chromedriver and some package.

Re: Beautiful Soup

#28

Post by jerf »

djtriptych wrote: post #8

Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…

"Might be less common now"

It is less necessary now. One of the most important parts of the HTML5 standards, IMHO, is that it specifies how to parse HTML that doesn't conform to the standards in a standard way. In principle, every bag of bytes now has a standard-compliant way to parse it that every HTML5 parser should agree on. I don't use this to know how many edge cases the standard and/or implementations have, but it's a lot better than it used to be, and means that every HTML5 parser has many of the capabilities that Beautiful Soup used to (nearly-)uniquely have for parsing messy HTML.

I suspect Beautiful Soup was a non-trivial aspect of how the decision to implement such a spec was decided upon. It proved the idea to be a very valuable one at a time when most languages lacked such a library. Basically, BS won so hard that while it wasn't necessarily directly adopted as a standard, the essence of it certainly was.

Re: Beautiful Soup

#29

Post by cogman10 »

djtriptych wrote: post #8

Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…

I work in finance and this thing is still indispensable for us. A LOT of financial info is only presented on a dynamically rendered HTML page that was last written in the bad old days.

Re: Beautiful Soup

#30

Post by toyg »

djtriptych wrote: post #8

Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…

> Might be less common now

That's because it's often deeply buried under more fashionable abstractions.

Post reply on HN