Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the best tools for web scraping in 2022?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 160 posts

Re: Ask HN: What are the best tools for web scraping in 2022?

#51

Post by namuorg »

I built a tool called Browserflow (https://browserflow.app) that lets you automate any task in the browser, including scraping websites.

People love it for its ease-of-use because you can record actions via click-and-point rather than having to manually come up with CSS selectors. It intelligently handles lists, infinite scrolling, pagination, etc. and can run on both your desktop and in the cloud.

Grateful for how much love it received when it launched on HN 8 months ago: https://news.ycombinator.com/item?id=29254147

Try it out and let me know what you think!

Re: Ask HN: What are the best tools for web scraping in 2022?

#52

Post by zarzavat »

Python is my work horse, if I need to scrape something from a site that is relaxed about scraping (most are). I have my own library of helper functions I've built up over the years. In simple cases I just regex out what I need, if I need a full DOM then I use JSDOM/node.

For sites that are "difficult" I remote control a real browser, GUI and all. I don't use Chrome headless because if there's e.g. a captcha I want to be able to fill it in manually.

Re: Ask HN: What are the best tools for web scraping in 2022?

#53

Post by 71a54xd »

Is there any form of markup / library that would allow me to access a file-tree similar to what show's up in the chrome "inspect" "sources" tab? I'm working on a system to extract m3u8 files from websites. Haven't found a good way to do this yet a few years since my last project that required scraping with a headless browser.

Re: Ask HN: What are the best tools for web scraping in 2022?

#54

Post by whoisjuan »

It depends on what you are trying to accomplish, but I think a combination of Puppeteer and JSDOM or Cheerio should take you far. Where it gets complex is when you need to do things such as rotating IPs, but in my experience, that's only needed if you're engaging in a heavy scraping workload.

Puppeteer + JSDOM is what I used to build https://www.getscrape.com, which is a high-level web scraping API. Basically, you tell the API if you want links, images, texts, headings, numbers, etc; and the API gets all that stuff for you without the need to pass selectors or parsing instructions.

In case anyone here wants something straightforward. It works well to build generic scraping operations.

Re: Ask HN: What are the best tools for web scraping in 2022?

#56

Post by samwillis »

I don’t think the landscape has changed much since then. However, from my experience you should do everything possible to avoid a headless browser for scraping. It’s in the region of 10-100x slower and significantly more resource intensive, even if you carefully block unwanted requests (images, css, video, ads).

Obviously sometimes you have to go that route.

Re: Ask HN: What are the best tools for web scraping in 2022?

#60

Post by traverseda »

Buttons840 wrote: post #27
altilunium wrote: post #20

Beautiful Soup gets the job done. I made several app by using it. [1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity) [2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia) [3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers) [4] https://github.com/altilunium/wi…

I've found lxml to be more powerful. The lxml library supports xpaths, which I don't believe Beautiful Soup does? In other words, consider lxml as well.

I reach for selectolax first if I'm doing relatively tame stuff. Also css selectors are nice.
Post reply on HN