I used Beautiful soup for a project recently (grabbing a series of page titles for export to CSV), it's super useful, thanks for the pointer OP.
Beautiful Soup
51–60 of 95 posts
Re: Beautiful Soup
#52Earlier quoted context omitted.
I’ve found Playwright to be a really great tool for scraping
This sounds interesting. Any resources for a beginner? I use Selenium regularly.
Playwright for Python has really good documentation: https://playwright.dev/python/
I used it for my https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
Re: Beautiful Soup
#53I used this recently to scrape a bunch of online restaurant reviews by a guy who I really like, use regex to get the postcode from the markup, do a geocode using postcodes.io, then plot the reviews on a Google map. It took about two / three hours and felt kinda dirty in a good way. Beautiful Soup made the first part really easy.
Hmm, cool project idea but if the subject doesn't know you're doing this, it is a bit weird to stalk their online footprint. Unless you mean the person is a professional restaurant reviewer and you like their opinion. Oh well, good luck either way.
Yes, they're a professional reviewer and I enjoy their reviews.
Re: Beautiful Soup
#54Beautiful Soup is an incredibly robust, and powerful tool. However, it can be sometimes be an intimidating tool for beginners (which ".find_x_y_z" method should I use again?). To that end you should check out gazpacho (with just a single "find" method): https://github.com/maxhumber/gazpacho
Re: Beautiful Soup
#55whats new about it? why is this post here? Beautiful Soup has been around for a long long time.
See: https://news.ycombinator.com/newsguidelines.html
Particularly: On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
Re: Beautiful Soup
#56Earlier quoted context omitted.
Because it's nice to exist outside the news cycle of what's "current" every once in a while :)
You have other social media for that. Low effort posts do not invite meaningful discussion and add moderation load.
Did you know that you don't have to read every article posted here, nor read every comment that is added?
If you don't want to participate in this post, it's OK to skip it. I skip dozens of posts a day - the best part is it's more efficient than going to them and putting the effort to whine!
As for not inviting meaningful discussion: there's some good discussion on this post - the very article you claim isn't capable of generating such.
Re: Beautiful Soup
#57I found lxml.html a lot easier to work with than bs4, in case that helps anyone else. https://lxml.de/lxmlhtml.html
Same here, I am unable to properly quantify it but there was something about the soup api I did not really like.
It may have been because I learned on the python xml.etree library in base(I moved to lxml because it has the same api but is faster and knows about parent nodes) and had a hard time with the soup api.
But I think it was the way it overloaded the selectors. I did not like the way you could magically find elements. I may have to revisit it and try and figure out why and if I still do not like it.
Re: Beautiful Soup
#58Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases. BS covered a LOT of these cases without forcing you to write t…
the jquery of backend
Re: Beautiful Soup
#59Earlier quoted context omitted.
I work in finance and this thing is still indispensable for us. A LOT of financial info is only presented on a dynamically rendered HTML page that was last written in the bad old days.
Absolutely. Until you have had to deal with this kind of problem first-hand , you have no idea how much of a relief it is that it exists. Sometimes, finding and using the right library can completely turn around a f'd project.
True.
And BS has been that library for me on at least 2 such projects.
Re: Beautiful Soup
#60I wrote my own HTML parser in Pascal 15 years ago. Pascal is much faster than Python