21–30 of 160 posts
I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc. The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.
In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer
Also is the ability to open the Networks panel, to snoop on requests and find the exact API call that you might need to perform your task, instead of having to pull in all of HTML/JS/CSS crap. As a lot of SPAs have essentially pushed everything behind JSON APIs, all information is usually one (authenticated) API call away.
I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc. The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.
It's been a while, but last time I used it Puppeteer had a headless Firefox backend?
scrapy is still king for me (scrapy.org). there are even packages to use headless browsers for those awful javascript heavy sites
however, APIs and RSS are still in play, and that does not require a heavy scraper. I am building vertical industry portals, and many of my data rollups consume APIs and structured XML/RSS feeds from social and other sites.
Beautiful Soup gets the job done. I made several app by using it. [1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity) [2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia) [3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers) [4] https://github.com/altilunium/wi…
In other words, consider lxml as well.
I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc. The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.
It's been a while, but last time I used it Puppeteer had a headless Firefox backend?
Selenium via Python is really useful too if you need to do a bit more (e.g clicks) than just fetching the html from the page.
Can I piggyback on the question and ask what are people scraping these days?
It also scrapes all my comments I upvoted and if those have links inside them it creates bookmarks from them too.
That's because I often find myself searching for some submission I upvoted but can't find it, especially if there were many similar ones, whereas Firefox bookmarks manager has a nifty search feature...
I had to scrape since the HN API doesn't expose ability to get information about upvoted submissions/comments. The extension assumes you are logged in, it doesn't ask for your username or password.
It's mostly for my own use and not very polished, but it works. I uploaded it to the Firefox extensions gallery, and you can probably find it there, but I don't think it's ready for public consumption yet...