111–120 of 160 posts
Has anybody created anything similar to Portia for scraping? I'd love to self host or pay a nominal fee to allow my team to create / adjust scrapers via a UI
Earlier quoted context omitted.
My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.
Yeah, my first step in trying to scrape an SPA is always to hit the network tab in the browser, filter by type=JSON and then sort by size. The largest responses are often the most useful, and can then be grabbed with curl. Sometimes though that's not enough - particularly on older sites that might use weirder concepts like ASP.NET View state. For those I find having Playwright around is a big benefit. Generally the t…
Puppeteer had it working, but if I need to do this in a google firebase function, it would be so much better to get it to simple fetch requests.
For the e-commerce sites I'm looking at though, most just are server side rendered, which is so much easier. In that case i use `cheerio` which has jQuery like syntax for crawling the raw text DOM.
As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control. Scrapers are very simple, effective and probably one of the least fun things to build.
Do you have recommendations for platforms that monitor scraping fleets? I occasionally code scrapers for quick data aggregation, but have trouble running anything long-term because it can be a chore to monitor. I've been looking into various options for self-hosting some sort of monitor/dashboard that can send alerts but haven't found anything satisfying yet.
Think about giving it a score based on how the data is shaped. If it's missing prices for example, then it immediately goes down to zero, doesn't update the database and sends an alert.
Earlier quoted context omitted.
My $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.
My £0.02 It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps. I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too. Alternatively sometimes doin…
Yikes.
In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices. [1] https://github.com/microsoft/playwright [2] https://github.com/puppeteer/puppeteer
If you're interested in running the puppeteer in containers, take a look at chrome-aws-lambda[1] and browserless docker container[2] Not affiliated with browserless, but they do have a free/paid cloud service. https://www.browserless.io [1] https://github.com/alixaxel/chrome-aws-lambda [2] https://github.com/browserless/chrome
Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes. The only thing I wish was present wa…
This comment got the same energy as this comment that dropbox could just be replaced by an FTP https://news.ycombinator.com/item?id=9224
one is signature / fingerprinting emulation. It helps to run the bot in a real browser and export the fingerprint (e.g. UA, canvass, geoloc etc) into JS object . Add noise to the data too.
Simulate residential IPs by routing through a residential proxy. If you run bots from cloud you will get blocked.