OCaml’s Lambda Soup (https://aantron.github.io/lambdasoup/) is a amazing library/, especially for those that prefer functional programming
The State of Web Scraping in 2021
41–50 of 132 posts
Re: The State of Web Scraping in 2021
#42Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API.
My favorite one implemented CSRF protections by polling an endpoint, and adding in the hashed data from that endpoint and a timestamp on every request.
When I hear a junior dev give up on something because the API doesn't provide the functionality of the UI, It makes me very sad that they're missing out.
Re: The State of Web Scraping in 2021
#43Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?
Selenium is famously unreliable, so a lot of people have been replacing it with headless chrome where they can.
I have been using selenium with chromedriver, I mistakenly thought those were basically the same thing.
Can you tell me more?
Re: The State of Web Scraping in 2021
#44Earlier quoted context omitted.
Selenium is famously unreliable, so a lot of people have been replacing it with headless chrome where they can.
Interesting. I was about to start on some web automation and so far I've had hammered into my head that Selenium is the 'language of the internet' or something along those lines. What would be a better solution, if you have any to recommend?
I'd suggest Puppeteer / Playwright. Both are great. Iirc the puppeteer team largely moved to playwright.
Re: The State of Web Scraping in 2021
#45Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
To be fair selenium style scraping can take a lot of time to setup if you aren’t already familiar with the tooling, and the browser rendering apis are unintuitive and sometimes flat out broken.
Re: The State of Web Scraping in 2021
#46Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?
> is it selenium replacement No completely different use case. Selenium is browser automation. Mechanical soup/Mechanize/Robobrowser are not actually web browsers, they have no javascript support either. They're python libraries that can simulate a web browser but doing GET requests, storing cookies across requests, filling http POST forms, etc. The downside is that they don't work with websites which rely on JavaScr…
I started reverse engineering web apis years ago as a more efficient way to scrape. Unfortunately there are new website builders (like salesforce lightning) that make really hard to recreate an http request due to the complexity of the parameters
Re: The State of Web Scraping in 2021
#47Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
To be fair selenium style scraping can take a lot of time to setup if you aren’t already familiar with the tooling, and the browser rendering apis are unintuitive and sometimes flat out broken.
That's why things like laravel's Dusk exists to put a layer over that complex experience.
Re: The State of Web Scraping in 2021
#48Earlier quoted context omitted.
Same as always - proxy farms, random popular UAs with random delays etc.
so will Google's freezing of the UA lead to less ability to web scrape for the non big company scrapers out there?
What? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape?
All Chrome is doing is stop appending the current semver in the UA it sends.
Re: The State of Web Scraping in 2021
#49any recommendation to scrape behind google social login?
Re: The State of Web Scraping in 2021
#50I've not followed this space. When I did, there were a lot of questions concerning the legality of automated scraping. Have those legal issues been resolved?