Beautiful Soup • Hacker News Board

Re: Beautiful Soup

#31

Post by mdaniel » Wed, Jun 08, 2022, 4:15 PM UTC

iooi wrote: post #25

I found lxml.html a lot easier to work with than bs4, in case that helps anyone else. https://lxml.de/lxmlhtml.html

On the off chance you were not aware, bs4 also supports[0] getting parse events from html5lib[1] which (as its name implies) is far more likely to parse the text the same way the browser would

0: https://www.crummy.com/software/BeautifulSoup/bs4/doc/index....

1: https://pypi.org/project/html5lib/

Re: Beautiful Soup

#32

Post by mdaniel » Wed, Jun 08, 2022, 4:17 PM UTC

cpach wrote: post #27

Does anyone know if there as a good equivalent for Go?

I've heard https://github.com/gocolly/colly#readme mentioned fondly, but I've never used it

Re: Beautiful Soup

#33

Post by move-on-by » Wed, Jun 08, 2022, 4:26 PM UTC

pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

I’ve found Playwright to be a really great tool for scraping

Re: Beautiful Soup

#34

Post by oh_sigh » Wed, Jun 08, 2022, 4:31 PM UTC

Boy...15 years ago I was reaching to this and hpricot almost every week to do some cool scraping/parsing of some kind. I always loved the bs API

Re: Beautiful Soup

#35

Post by zasdffaa » Wed, Jun 08, 2022, 4:39 PM UTC

For web scraping I used htmltidy (https://en.wikipedia.org/wiki/HTML_Tidy) which cleaned it sufficiently that I could run XSLT over it (gags at the memory)

Re: Beautiful Soup

#36

Post by cyberlurker » Wed, Jun 08, 2022, 4:44 PM UTC

frereubu wrote: post #20

I used this recently to scrape a bunch of online restaurant reviews by a guy who I really like, use regex to get the postcode from the markup, do a geocode using postcodes.io, then plot the reviews on a Google map. It took about two / three hours and felt kinda dirty in a good way. Beautiful Soup made the first part really easy.

Hmm, cool project idea but if the subject doesn't know you're doing this, it is a bit weird to stalk their online footprint.

Unless you mean the person is a professional restaurant reviewer and you like their opinion.

Oh well, good luck either way.

Re: Beautiful Soup

#37

Post by 867-5309 » Wed, Jun 08, 2022, 4:50 PM UTC

werds wrote: post #11

phone8675309 wrote: post #7

How helpful is this when you're dealing with a website that does not degrade gracefully and insists using JavaScript to shove things in where a static webpage would work? (For example, scraping football scores from NFL.com)

its not useful in those cases, but usually for those js rendered sites you can replicate the ajax requests which happen and get nicely formed json documents to parse through instead.

nowadays you usually have to submit http headers and cookies too, that's always a fun process of elimination

Re: Beautiful Soup

#38

Post by derac » Wed, Jun 08, 2022, 4:50 PM UTC

move-on-by wrote: post #33

pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

I’ve found Playwright to be a really great tool for scraping

Playwright is easily the best for browser automation. I still use requests + beautiful soup often as well.

Re: Beautiful Soup

#39

Post by samwillis » Wed, Jun 08, 2022, 4:57 PM UTC

djtriptych wrote: post #8

Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…

Have literally been building something with BS today. It’s very much still a current library, I imagine in some areas people have moved on but I will continue to reach for it.

Re: Beautiful Soup

#40

Post by samwillis » Wed, Jun 08, 2022, 5:01 PM UTC

move-on-by wrote: post #33

pmoriarty wrote: post #16

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

I’ve found Playwright to be a really great tool for scraping

Headless scraping is in the region of 10x slower and more resource intensive, even when carefully blocking requests such as images. It should always be a second choice.

Other than that Playwright is incredible, by far the best browser automation api.