Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

31–40 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#31

Can I piggyback on the question and ask what are people scraping these days?

During Covid lockdown online food shopping in my area (UK) became increasingly difficult to book delivery slots as they would fill up as soon as they became available. For most people who either owned a car or lived nearby a supermarket this was probably just a mild inconvenience, however at the time I didn't have a car and the nearest store was miles away from my house. Writing a bunch of Python scripts to check for free delivery slots over several of the big supermarkets and then pinging me when one became available was a lifesaver (maybe a bit dramatic, but it helped me secure regular food deliveries without having to spend hours hitting the refresh button). I learned a lot in the process, the lengths at which some online stores go to try and deter automated scripts was quite interesting.

Re: Ask HN: What are the best tools for web scraping in 2022?

#33
post #31

Can I piggyback on the question and ask what are people scraping these days?

During Covid lockdown online food shopping in my area (UK) became increasingly difficult to book delivery slots as they would fill up as soon as they became available. For most people who either owned a car or lived nearby a supermarket this was probably just a mild inconvenience, however at the time I didn't have a car and the nearest store was miles away from my house. Writing a bunch of Python scripts to check for…

I feel you. I've wondered many times why anti-bot scripts are deployed at all, save some cases like airline booking, where airlines are penalized for abnormal look-book ratios.

Re: Ask HN: What are the best tools for web scraping in 2022?

#34

Beautiful Soup gets the job done. I made several app by using it. [1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity) [2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia) [3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers) [4] https://github.com/altilunium/wi…

I've found lxml to be more powerful. The lxml library supports xpaths, which I don't believe Beautiful Soup does? In other words, consider lxml as well.

lxml is supported (mostly) out of the box for BeautifulSoup, so you can it as a parser behind BS4's nicer interface, which I believe the OP does in the linked codebases.

Re: Ask HN: What are the best tools for web scraping in 2022?

#35
post #3

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer

Highly recommend playwright (if I'm not mistaken most of the big developers from puppeteer were hired by MS to work on playwright). I run into significantly less await/async problems with playwright than I did with puppeteer and the codegen tool is super helpful as a first pass option.

Re: Ask HN: What are the best tools for web scraping in 2022?

#39
For simple scraping where the content is fairly static, or when performance is critical, I will use linkedom to process pages.

https://github.com/WebReflection/linkedom

When the content is complex or involves clicking, Playwright is probably the best tool for the job.

https://github.com/microsoft/playwright

Post reply on HN