Can I piggyback on the question and ask what are people scraping these days?
Ask HN: What are the best tools for web scraping in 2022?
31–40 of 160 posts
Re: Ask HN: What are the best tools for web scraping in 2022?
#32Re: Ask HN: What are the best tools for web scraping in 2022?
#33Can I piggyback on the question and ask what are people scraping these days?
During Covid lockdown online food shopping in my area (UK) became increasingly difficult to book delivery slots as they would fill up as soon as they became available. For most people who either owned a car or lived nearby a supermarket this was probably just a mild inconvenience, however at the time I didn't have a car and the nearest store was miles away from my house. Writing a bunch of Python scripts to check for…
Re: Ask HN: What are the best tools for web scraping in 2022?
#34Beautiful Soup gets the job done. I made several app by using it. [1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity) [2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia) [3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers) [4] https://github.com/altilunium/wi…
I've found lxml to be more powerful. The lxml library supports xpaths, which I don't believe Beautiful Soup does? In other words, consider lxml as well.
Re: Ask HN: What are the best tools for web scraping in 2022?
#35In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer
Re: Ask HN: What are the best tools for web scraping in 2022?
#36Re: Ask HN: What are the best tools for web scraping in 2022?
#37Re: Ask HN: What are the best tools for web scraping in 2022?
#38[1] https://github.com/brutuscat/medusa-crawler
Which I maintain as a fork of the unmaintained Anemone gem.
Re: Ask HN: What are the best tools for web scraping in 2022?
#39https://github.com/WebReflection/linkedom
When the content is complex or involves clicking, Playwright is probably the best tool for the job.
Re: Ask HN: What are the best tools for web scraping in 2022?
#40Can I piggyback on the question and ask what are people scraping these days?