The State of Web Scraping in 2021 • Hacker News Board

Re: The State of Web Scraping in 2021

#21

Post by numeralls » Mon, Oct 11, 2021, 3:47 PM UTC

amelius wrote: post #11

> Crawl at off-peak traffic times. If a news service has most of its users present between 9 am and 10 pm – then it might be good to crawl around 11 pm or in the wee hours of the morning. How do you know this if it is not your website? Also, the internet has no time zone.

If your scraping a popular website Google Trends should be a pretty good proxy

Re: The State of Web Scraping in 2021

#22

Post by hdfinenwvdun » Mon, Oct 11, 2021, 3:48 PM UTC

bryanrasmussen wrote: post #15

hdjjhhvvhga wrote: post #4

Earlier quoted context omitted.

Same as always - proxy farms, random popular UAs with random delays etc.

so will Google's freezing of the UA lead to less ability to web scrape for the non big company scrapers out there?

You can easily spoof the UA

Re: The State of Web Scraping in 2021

#23

Post by gizdan » Mon, Oct 11, 2021, 3:50 PM UTC

Surprised Woob[0] (formerly Weboob) isn't on the list. It's designed for specific tasks, such as getting transactions from your bank, events from different event pages, and much.

[0] https://woob.tech/

Re: The State of Web Scraping in 2021

#24

Post by nathell » Mon, Oct 11, 2021, 3:51 PM UTC

I’ll chime in with mine: Skyscraper (Clojure) [0] builds on Enlive/Reaver (which in turn build on JSoup), but tries to address cross-cutting concerns like caching, fetching HTML (preferably in parallel), throttling, retries, navigation, emitting the output as a dataset, etc.

Re: The State of Web Scraping in 2021

#25

Post by m_ke » Mon, Oct 11, 2021, 3:55 PM UTC

Another tip, there are a few browser extensions that can record your interactions and generate a playwright script.

Here's one: https://chrome.google.com/webstore/detail/headless-recorder/...

Re: The State of Web Scraping in 2021

#26

Post by conradfr » Mon, Oct 11, 2021, 4:00 PM UTC

zlib wrote: post #14

What kind of stuff are people needing to scrape?

I have a side-project where I display the schedule of the day of 100+ French radios, like you would for TV channels.

Scraping works great to get the data.

I don't like node/js but I use it to do the scraping as I view the code as trash and full of edge cases and unreliable data / types and I can't complain, a dynamic scripting language is great for that.

Re: The State of Web Scraping in 2021

#27

Post by throwawaysea » Mon, Oct 11, 2021, 4:03 PM UTC

Is there open source software that can extract the "content" part of a given page cleanly? I'm thinking about what the reader mode in browsers can do as an example, where the main content is somehow isolated and displayed.

Re: The State of Web Scraping in 2021

#28

Post by f311a » Mon, Oct 11, 2021, 4:06 PM UTC

For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster.

Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright.

It looks like the author of the article just googled some libraries for each language and didn't research the topic.

Re: The State of Web Scraping in 2021

#29

Post by gcatalfamo » Mon, Oct 11, 2021, 4:06 PM UTC

Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?

Re: The State of Web Scraping in 2021

#30

Post by benzible » Mon, Oct 11, 2021, 4:09 PM UTC

I just needed a service to reliably fetch raw pages that I can process in my own application and so far I've been happy with this: https://promptapi.com/marketplace/description/adv_scraper-ap...

$30 / month for 300K requests, rotating residential proxies, uses headless Chromium, etc.