I'm in the web scraping industry for a while and I often spend some time creating my "swiss knife" with Playwright or Selenium in case things get tough.
Thanks to a niche substack I'm following, I discovered only today the existence of anti detect browsers like GoLogin and others.
From what I see, they seem a good solution for small projects, but difficult to scale in larger ones for costs of licensing and infrastructure (most of them require a windows machine to run).
Does any of you guys smarter than me use these browsers on a large scale? How is composed your tech stack?
Ask HN: Have you ever used anti detect browsers for web scraping?
1–10 of 27 posts
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#2No post body was provided.
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#3No post body was provided.
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#4Can you share the substack?
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#5this smells like an ad for GoLogin
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#6Can you share the substack?
I didn’t mention because someone would think I’m promoting it: it’s called The Web Scraping Club.
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#7this smells like an ad for GoLogin
In the substack post are mentioned several browsers, so I would say no.
I didn’t even mention the Substack so no one can say I’m promoting it.
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#8Can you share the substack?
I didn’t mention because someone would think I’m promoting it: it’s called The Web Scraping Club.
Seems like a fairly shallow (content marketing fluff) substack to be honest. There are better places to follow, like: https://www.trickster.dev/post/
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#9How would you actually use an anti-detect browser programmatically? Would you need to write a custom Selenium driver for it or equivalent for Playwright? Even if the browser is built off something like Chrome, you'd still need a way to interact with the anti-detect related features.
A good trick I discovered is using webkit thru Playwright to bypass fingerprinting and related anti-bot measures. Firefox/Chrome simply leaks too much information, even with various "stealth" modifications. e.g: have been able to reliably scrape a well known companies site that implemented a "state of the art, AI-powered, behavioral analysis, etc" anti-bot product. Using Chrome/Firefox + stealth measures in Playwright did not work - simply switching to Webkit with no further modifications did the trick.
Not exactly what you're asking, but my point is, that with a little time and effort, I've usually been able to find fairly simple holes in most anti-bot measures -- it probably wouldn't be terribly hard (especially since you're versed in scraping) to build-out something similar to what you're looking to achieve without having to pay for sketchy anti-detect browsers.
Re: Ask HN: Have you ever used anti detect browsers for web scraping?
#10How would you actually use an anti-detect browser programmatically? Would you need to write a custom Selenium driver for it or equivalent for Playwright? Even if the browser is built off something like Chrome, you'd still need a way to interact with the anti-detect related features. A good trick I discovered is using webkit thru Playwright to bypass fingerprinting and related anti-bot measures. Firefox/Chrome simply…
Any tips/code examples for your webkit solution(s)? Where does one begin with using webkit for scraping?
I think using anti-fingerprinting is itself a fingerprint. I imagine it would be easier to hide in the noise of regular browsers.