11–20 of 95 posts
How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)
How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)
How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)
Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…
How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)
IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else.
I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.
Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.
How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)
its not useful in those cases, but usually for those js rendered sites you can replicate the ajax requests which happen and get nicely formed json documents to parse through instead.
For example, the new Google Play Store website stores the data in AF_initDataCallback calls and can be extracted with re.findall(r"AF_initDataCallback\((.*?)\);", html_string).