11–20 of 96 posts
I've used XPath for crawling with selenium, and it used to be my favorite way, but turned out quite unreliable if you don't combine it with other selectors as certain website are really badly designed and have no good patterns. So what's the added value over pure selenium?
Earlier quoted context omitted.
I am quite aware, but I actually built most of the scraping logic a long time ago, before I even knew that playwright was a thing. I am looking to refactor a lot of this, and switching over to playwright is a high priority, using something like camoufox for scraping, instead of just chromium. Most of my work on this the past month has been simple additions that are nice to haves
I was in a similar boat with my scrapers. Started with Selenium 5-6 years ago and only discovered Playwright 2 years ago. Spend a month or so swapping the two, which was well worth it. Cleaner API, async support.
Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
With the custom headers, you can actually trick a lot of sites with bot protection to let you load their sites (even big sites like youtube, which I have found success in)
Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
[flagged]
You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool:
- Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string.
This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_header#Format_for_a...>. Many sites mention in public API guidelines to include an email address so you can be contacted in case of problems. If you don't include this and you're causing trouble, all I can do is ban your IP address altogether (or entire ranges: if you hop between several IPs I'll have to assume you have access to the whole range). Nobody likes IP bans: you have to get a new IP, your provider has a burned IP address, the next customer runs into issues... don't be this person, include an identifier.
- Timing out the request after a few seconds.
Some pages on my site involve number crunching and take 20 seconds to load. I could add complexity to do this async instead, but, by having it live, the regular users get the latest info and they know to just wait a few seconds and everybody is happy. Even the scrapers can get the info, I'm fine computing those pages for you. But if you ask for me to do work and then walk away, that's just rude. It shows up in my logs as HTTP status 499 and I'll ban scrapers that I notice doing this regularly
- Ignoring robots.txt.
I have exactly 1 entry in there, and that's a caching proxy for another site that is struggling with load. If you ignore the robots file and just crawl the thing from A to Z at a high rate, that causes a lot of requests to the upstream site for updating stale caches. You can obviously expect a ban because it's again just a waste of resources
I used to scrape back in the day when it was easy (literally just make a request and parse html). Seems cloudflare checkboxes / human verification are very commonplace nowdays. Curious how(/if) web scrapers get around those?