61–70 of 143 posts
I think there might be a market for "human crawlers". Just like people use Mechanical Turk to get humans to beat CAPTCHAs, you could use it to get humans to visit a web page for you, and return its HTML source. There are of course residential proxy services (ie HolaVPN), but they're still technically can be detected.
Why would you do that when you can automate it?
Those are averages of multiple runs on a Core i7-8550U running Chromium 75.0.3770.90 on Ubuntu 19.04.
isNan and isFinite are much slower in headless mode, but other functions like parseFloat and parseInt aren't. My guess is that the backend is comparing the relative times that certain functions take. If isNan and isFinite take the same time as parseFloat, then you're not in headless mode. If those functions take 6x longer than parseFloat, you're in headless mode.
I don't know if this holds true for non x86 architectures or other platforms.
Well the user agent of chrome headless contains 'HeadlessChrome' according to this site [1]. Sure enough when I spoof my user agent to the first in the list it magically determines I'm using headless Chrome. He basically says he's inspecting user agents: > Under the hood, I only verify if browsers pretending to be Chromium-based are who they pretend to be. Thus, if your Chrome headless pretends to be Safari, I won’t…
If a browser claims to be Headless Chrome, you believe it. Nobody has a reason to lie about that. The interesting question is the opposite case: is somebody claiming to be a normal Chrome, but is actually Headless Chrome (or an automated member of some other browser family, or not a browser at all but e.g. a Python script). So if you take a Headless Chrome instance but change the User-Agent to match that of a normal…
I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
If you do want to stick with Selenium, you're better studying the chromedriver source than Chromium itself.
I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
Have you tried loading a full browser session? Not just headless.
I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user.
Where do you run your scraping operation? Is it in AWS or some other hosting provider, because that will get you blocked quickly by a lot of sites? Do you rate limit, including adding random jitter to mimic the way a human might use a browser?
There's scraping services available that essentially use a network of browsers on residential connections with their extension installed to get around scraping detection. It's much slower, but it's much more reliable. We also had some success by signing up with a bunch of the VPN providers (PIA, NordVPN, ExpressVPN, etc) and cycling through their servers frequently. Anything to avoid creating patterns that look automated or being tied to an IP that can be blacklisted. I'd start there before I'd worry about hacky javascript detection like in this story being what's tripping you up.
https://arh.antoinevastel.com/javascripts/fpCollect.min.js contains some functions called redPill that aren't in the normal fpCollect library. redPill3 measures the time of some JS functions and sends that data to the backend. Here's a chart of redPill3's timing data on my computer: https://i.imgur.com/c8iuV6I.png Those are averages of multiple runs on a Core i7-8550U running Chromium 75.0.3770.90 on Ubuntu 19.04. is…
Unexpectedly, it turned out that Accept header was perfect for this. The final chart was this:
https://i.imgur.com/ZA8qD8t.png
("link" means clicking on an URL or entering it manually; "embedded" means tag)
Makes me wonder whether Accept header is still useful for fingerprinting in general, and distinguishing between headless and headful(?) browsers in particular.
Earlier quoted context omitted.
Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.
Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does…
Earlier quoted context omitted.
I don't empathize with your viewpoint because, whether it's a web scraper, or a person, the work is exactly the same. There's no additional volume, or extra steps. We just emulate a worker. We measure the value in FTEs, and when a researcher quits, we do not replace them if the appropriate FTEs have been reached with projects. It's a major benefit to the business not only because we don't have to pay another employee…
Sadly, you are correct to have realized that many posters on HN are so naive that they will offer you $0/hour consulting for your for-profit business. Posting on the HN forums means you "don't have to pay another employee" that's an expert in the field. I can't do much to prevent this, but I don't much respect it, either.
If it reduces market for some consultants, well, sucks to be them, they'd better find a different way of providing value. Not every value needs to be captured and priced. A world in which all value was captured and priced would really suck.
Wouldn't these render the Brave browser unusable on some sites?
