Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping in 2021

mihaisplace.blog

Post reply on HN
Print view View original

61–70 of 132 posts

Re: The State of Web Scraping in 2021

#61

Post by kjkjadksj »

xzel wrote: post #44
melomal wrote: post #40

Earlier quoted context omitted.

Interesting. I was about to start on some web automation and so far I've had hammered into my head that Selenium is the 'language of the internet' or something along those lines. What would be a better solution, if you have any to recommend?

I'd suggest Puppeteer / Playwright. Both are great. Iirc the puppeteer team largely moved to playwright.

Puppeteer is frustrating to me. When I tried to use it I couldn’t get it to click buttons, but I did get it to hover on the button so I know I had the correct element in my code. Their click function just did nothing at all. I resorted to tabbing a certain amount of times and hitting enter.

Re: The State of Web Scraping in 2021

#62

Post by sidharthv »

m_ke wrote: post #25

Another tip, there are a few browser extensions that can record your interactions and generate a playwright script. Here's one: https://chrome.google.com/webstore/detail/headless-recorder/...

If you don't want to install another extension, Playwright has built in support for recording.

npx playwright codegen wikipedia.org

https://playwright.dev/docs/next/codegen

Re: The State of Web Scraping in 2021

#63

Post by jrochkind1 »

m_ke wrote: post #25

Another tip, there are a few browser extensions that can record your interactions and generate a playwright script. Here's one: https://chrome.google.com/webstore/detail/headless-recorder/...

I'm not familiar with "playwright", it doesn't seem to be mentioned in OP either.

When I google, I see it advertised as a "testing" tool.

Can I also use it for scraping? Where would I learn more about doing so?

Re: The State of Web Scraping in 2021

#64

Post by jrochkind1 »

jmnicolas wrote: post #3

Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium. I wonder what does a scraping framework offer?

Just for one example, when you have to get a form, and then submit the form, with the CSRF protection that was in the form... of course you COULD write that yourself by printing HTML and then analyzing it with C# (which triggers more requests to chromium I guess), but you're probably going to wonder why you are reinventing the wheel when you want to be getting on to the domain-specific stuff.

Re: The State of Web Scraping in 2021

#65

Post by rmetzler »

jrochkind1 wrote: post #63
m_ke wrote: post #25

Another tip, there are a few browser extensions that can record your interactions and generate a playwright script. Here's one: https://chrome.google.com/webstore/detail/headless-recorder/...

I'm not familiar with "playwright", it doesn't seem to be mentioned in OP either. When I google, I see it advertised as a "testing" tool. Can I also use it for scraping? Where would I learn more about doing so?

Playwright is similar to Puppeteer, but can use different browsers not only Chrome.

Re: The State of Web Scraping in 2021

#68

Post by jmnicolas »

jrochkind1 wrote: post #64
jmnicolas wrote: post #3

Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium. I wonder what does a scraping framework offer?

Just for one example, when you have to get a form, and then submit the form, with the CSRF protection that was in the form... of course you COULD write that yourself by printing HTML and then analyzing it with C# (which triggers more requests to chromium I guess), but you're probably going to wonder why you are reinventing the wheel when you want to be getting on to the domain-specific stuff.

Ah yes I see. Mine was read-only, so no need for complex stuff.

Re: The State of Web Scraping in 2021

#69

Post by jmnicolas »

Veen wrote: post #5
jmnicolas wrote: post #3

Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium. I wonder what does a scraping framework offer?

> I wonder what does a scraping framework offer? HTTP requests, HTML parsing, crawling, data extraction, wrapping complex browser APIs etc. Nothing you couldn't do yourself, but like most frameworks, they abstract the messy details so you can get a scraper working quickly without having to cobble together a bunch of libraries or re-invent the wheel.

I see thanks.

Re: The State of Web Scraping in 2021

#70

Post by duckmysick »

gcatalfamo wrote: post #29

Why no mention of selenium? Is it not cool anymore? I have never heard of mechanicalsoup: is it selenium replacement?

I moved from selenium to playwright. It has a pleasant API and things just works our of box. I ran into odd problems with selenium before, especially when waiting for a specific element. Selenium didn't register it, but I could see it load.

It was uncharacteristic of me, because I tend to use boring, older technologies. But this gamble paid off for me.

https://playwright.dev/

Post reply on HN