The State of Web Scraping in 2021

#1

Post by marvram » Mon, Oct 11, 2021, 12:21 PM UTC

The State of Web Scraping in 2021
mihaisplace.blog

Re: The State of Web Scraping in 2021

#2

Post by juanse » Mon, Oct 11, 2021, 2:00 PM UTC

Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.

Re: The State of Web Scraping in 2021

#3

Post by jmnicolas » Mon, Oct 11, 2021, 2:01 PM UTC

Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#.

I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium.

I wonder what does a scraping framework offer?

Re: The State of Web Scraping in 2021

#4

Post by hdjjhhvvhga » Mon, Oct 11, 2021, 2:07 PM UTC

juanse wrote: post #2

Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.

Same as always - proxy farms, random popular UAs with random delays etc.

Re: The State of Web Scraping in 2021

#5

Post by Veen » Mon, Oct 11, 2021, 2:09 PM UTC

jmnicolas wrote: post #3

Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium. I wonder what does a scraping framework offer?

> I wonder what does a scraping framework offer?

HTTP requests, HTML parsing, crawling, data extraction, wrapping complex browser APIs etc. Nothing you couldn't do yourself, but like most frameworks, they abstract the messy details so you can get a scraper working quickly without having to cobble together a bunch of libraries or re-invent the wheel.

Re: The State of Web Scraping in 2021

#6

Post by juanse » Mon, Oct 11, 2021, 2:11 PM UTC

hdjjhhvvhga wrote: post #4

juanse wrote: post #2

Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.

Same as always - proxy farms, random popular UAs with random delays etc.

Sorry, what is a UA?

Re: The State of Web Scraping in 2021

#7

Post by shmoogy » Mon, Oct 11, 2021, 2:12 PM UTC

juanse wrote: post #6

hdjjhhvvhga wrote: post #4

Earlier quoted context omitted.

Same as always - proxy farms, random popular UAs with random delays etc.

Sorry, what is a UA?

User Agent

Re: The State of Web Scraping in 2021

#8

Post by Veen » Mon, Oct 11, 2021, 2:12 PM UTC

juanse wrote: post #6

hdjjhhvvhga wrote: post #4

Earlier quoted context omitted.

Same as always - proxy farms, random popular UAs with random delays etc.

Sorry, what is a UA?

User agent (browser or some other web client).

Re: The State of Web Scraping in 2021

#9

Post by colinramsay » Mon, Oct 11, 2021, 2:13 PM UTC

If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a config file:

https://gotripod.com/insights/super-simple-site-crawling-and...

[1] http://go-colly.org/

Re: The State of Web Scraping in 2021

#10

Post by Ozzie_osman » Mon, Oct 11, 2021, 2:33 PM UTC

juanse wrote: post #2

Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.

Fundamentally, what most scrapers learn is that the more their scraper can behave like a human browsing the site, the less likely they are to get detected and blocked.

This does put limits on how quickly they can crawl, of course, but scrapers find ways around it like changing ip and user agent (ip is probably the main one, bec you can then pretend that you are multiple humans browsing the site normally).