Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping in 2021

mihaisplace.blog

Post reply on HN
Print view View original

81–90 of 132 posts

Re: The State of Web Scraping in 2021

#81

Post by novaleaf »

lloydatkinson wrote: post #77
novaleaf wrote: post #76

Self promotion: my SaaS is the lowest cost web scraping tool for high volume, and has been in business since 2016. https://PhantomJsCloud.com My SaaS requires some technical knowledge to use (call a web api) which I suppose is why it's not ever in these lists. Some of my customers are *very* large businesses. If you are looking at evading bot countermeasures, my product isn't probably the best for you. but for TCO no…

Isn't phantomjs deprecated and unmaintained?

yes, bad naming on my part. While it does support PhantomJs still, the default is a Puppeteer backend.

Re: The State of Web Scraping in 2021

#82

Post by tannhaeuser »

throwawaysea wrote: post #27

Is there open source software that can extract the "content" part of a given page cleanly? I'm thinking about what the reader mode in browsers can do as an example, where the main content is somehow isolated and displayed.

You can use SGML (on which HTML is/was based) and my LGPL-licensed sgmljs package [1] for that, plus my SGML DTD grammar for HTML5. [2] describes common tasks in preservation of Web content to give you a flavor, but you can customize what SGML does with your markup to death really; in your case, you'll probably want to throw away divs and navs to get clean semantic HTML which you can do using SGML link processes (= pipeline of markup filters and transformations), but you could also convert HTML into canonical markup (eg XML) and use Turing-complete XML processing tools such as XSLT as described in the linked tutorial.

[1]: http://sgmljs.net

[2]: http://sgmljs.net/docs/parsing-html-tutorial/parsing-html-tu...

Re: The State of Web Scraping in 2021

#83

Post by xnyan »

jrochkind1 wrote: post #63
m_ke wrote: post #25

Another tip, there are a few browser extensions that can record your interactions and generate a playwright script. Here's one: https://chrome.google.com/webstore/detail/headless-recorder/...

I'm not familiar with "playwright", it doesn't seem to be mentioned in OP either. When I google, I see it advertised as a "testing" tool. Can I also use it for scraping? Where would I learn more about doing so?

Playwright is essentially a headless chrome, firefox, and webkit browser with a nice API that's intended for automation/scraping. It's far more heavy than something like curl, but it has all the capabilities of any browser you want (not just chrome as with puppeteer) and makes stuff like interacting with javascript a breeze.

It's similar to Google's puppeteer, but in my opinion even with chrome much more pleasant and productive. Microsoft's best developer tool IMO, saves me tons of time.

Re: The State of Web Scraping in 2021

#84

Post by xnyan »

f311a wrote: post #28

For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright. It looks like the author of the article just googled some libraries for each language and didn't research the topic.

I agree with your conclusion, but in any discussion about web scraping it's probably a good idea to mention BeautifulSoup given how popular it is (virtually a builtin in terms how much it's used) and given all the documentation available for it, a good starting point if perf is not going to be a concern.

Re: The State of Web Scraping in 2021

#85

Post by eastendguy »

dec0dedab0de wrote: post #42

Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…

> Scraping things that don't want to be scraped

If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.

Re: The State of Web Scraping in 2021

#86

Post by colinramsay »

hivacruz wrote: post #32
colinramsay wrote: post #9

If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a config file: https://gotripod.com/insights/super-simple-site-crawling-and... [1] http://go-colly.org/

I used this library to get familiar with Go. It is indeed very powerful and really easy to create a scraper. My main concerns though were about testing. What if you want to create tests to check if your scraper still gets the data we want? Colly allows nested scraping and it's easy to implement but you have all your logic into one big function, making it harder to test. Did you find a solution to this? I'm considerin…

Not yet but my plan was to just have a static HTML site which the tests could run against.

Re: The State of Web Scraping in 2021

#87

Post by mro_name »

I am scraping radio broadcast pages for a decade now. Started with (ruby) scrapy, then nokogiri, then moved on to go and their html package.

Currently sport a mix of curl + grep + xsltproc + lambdasoup (OCaml) and am happy with it. Sounds like a mess but is shallow, inspectable, changeable and concise. http://purl.mro.name/recorder

Re: The State of Web Scraping in 2021

#90

Post by stef25 »

specproc wrote: post #37
throwawaysea wrote: post #27

Is there open source software that can extract the "content" part of a given page cleanly? I'm thinking about what the reader mode in browsers can do as an example, where the main content is somehow isolated and displayed.

I believe the main library for reader mode is called readability. I played around with a python implementation a while back. Just pipe in your raw html as part of the process. It's good, but not flawless. If I remember correctly, it included some quotes and image text as part of the body for the site I tried it on.

There's a PHP port of Readability and it works for some sites, for others not at all. Very far from perfect.
Post reply on HN