Can I piggyback on the question and ask what are people scraping these days?
US Treasury rates
Ask HN: What are the best tools for web scraping in 2022?
61–70 of 160 posts
Re: Ask HN: What are the best tools for web scraping in 2022?
#62It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand.
I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems.
I wrote my own CLI scraping tool on top of Playwright a few months ago, which has been a fun way to explore Playwright's capabilities: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
Re: Ask HN: What are the best tools for web scraping in 2022?
#63Quick plug for running scrapers in GitHub Actions and writing the results back to a repository - which gives you a free way to track changes to a scraped resource over time. I call this "Git scraping" - I've written a whole bunch of notes about this technique here: https://simonwillison.net/series/git-scraping/
Re: Ask HN: What are the best tools for web scraping in 2022?
#64Scrapy for low level stuff plus some tool that can run Javascript
Re: Ask HN: What are the best tools for web scraping in 2022?
#65Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…
Also a fan. Usually I generate indexes/urls, and then just wget and scrape the content offline once all is downloaded.
Re: Ask HN: What are the best tools for web scraping in 2022?
#66Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…
This comment got the same energy as this comment that dropbox could just be replaced by an FTP https://news.ycombinator.com/item?id=9224
Re: Ask HN: What are the best tools for web scraping in 2022?
#67if you are tech enough to find the query-selector of the elements, here is a great tool.
Great things is, it have support for Zapier, webhooks and API access too.!
Re: Ask HN: What are the best tools for web scraping in 2022?
#68e-scraper.com
Re: Ask HN: What are the best tools for web scraping in 2022?
#69I built a tool called Browserflow ( https://browserflow.app ) that lets you automate any task in the browser, including scraping websites. People love it for its ease-of-use because you can record actions via click-and-point rather than having to manually come up with CSS selectors. It intelligently handles lists, infinite scrolling, pagination, etc. and can run on both your desktop and in the cloud. Grateful for how…
Browserflow fan here - I used it on a variety of projects. DK was a great help throughout the way on the slack group.
It's fun to watch his twitter and celebrate his wins alongside him
Re: Ask HN: What are the best tools for web scraping in 2022?
#70It’s still curl and selenium.