Detecting Chrome headless, the game goes on • Hacker News Board

Re: Detecting Chrome headless, the game goes on

#51

Post by Pinbenterjamin » Fri, Jul 19, 2019, 5:55 PM UTC

slaymaker1907 wrote: post #48

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I personally think that it maybe be ethically questionable to be making background checks easier. There is a reason why the right to be forgotten is becoming a thing in various jurisdictions and lack of easy access for sensitive data is one countermeasure to try and counterbalance the need for public access to data with the right to privacy for individuals.

I don't have a perspective on the ethics of easier background checks. We run employment checks, the ultimate decision of whether to hire falls to the customer ALWAYS. I've seen plenty of former criminals get hired. It's a workplace culture 'thing'.

The right to be forgotten is alive and well most of the time, 90% of our clients don't observe information further back than a few years. I feel like that is a fair assessment of someone's behavior.

Re: Detecting Chrome headless, the game goes on

#52

Post by elorant » Fri, Jul 19, 2019, 6:02 PM UTC

paganel wrote: post #45

elorant wrote: post #40

Earlier quoted context omitted.

Have you tried loading a full browser session? Not just headless.

Not the OP, but I did that about 12 years ago, with Firefox. My boss at the time had asked me to parse some public institution website that was quite difficult to write a parser for directly in Python, so in the end we just decided to write a quick extension for Firefox and let an instance of it run on a spare computer. That public institution website had some JS bug that would cause FF to gobble up memory pretty fas…

Sure we do. Through Selenium. You can either load a full browser session, or a headless one. But headless sessions are identifiable.

Re: Detecting Chrome headless, the game goes on

#53

Post by skybrian » Fri, Jul 19, 2019, 6:05 PM UTC

ryandrake wrote: post #21

Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Intern…

This is like saying, if you're going to give people free samples, why not give away the whole grocery store?

Wanting to give out limited free samples inevitably leads to making sure you are giving out samples to people and not bots and not too much to each person, and that leads to user tracking.

Compare with the arms race between newspapers and incognito mode:

https://www.blog.google/outreach-initiatives/google-news-ini...

Re: Detecting Chrome headless, the game goes on

#54

Post by ijpoijpoihpiuoh » Fri, Jul 19, 2019, 6:08 PM UTC

floatingatoll wrote: post #42

stickfigure wrote: post #38

Earlier quoted context omitted.

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form. In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

If such a national law were passed with funding guaranteed for open publication of records, I would endorse your point of view. No such funding exists, and municipalities are regularly denied tax increases by their voters for any reason — much less public records publication that would often embarrass and humiliate those same voters. So in essence you're asking them to cut public services and staffing in order to giv…

Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.

Re: Detecting Chrome headless, the game goes on

#55

Post by ijpoijpoihpiuoh » Fri, Jul 19, 2019, 6:09 PM UTC

huhtenberg wrote: post #41

stickfigure wrote: post #38

Earlier quoted context omitted.

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form. In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

Public records are published based on certain demand assumptions. If a real-world demand for, say, some GIS data is hundreds of requests per day, then a crawler that comes in with hundreds requests PER MINUTE will obviously stress the infrastructure. Adjusting infrastructure to cope is not an instant process, nor is it a sure thing to begin with given all the budgeting formalities. So your "civic duty" will ultimatel…

You've made an unfounded assumption -- that is, that the person you're responding to is scraping irresponsibly. If they are, as they say, simply replacing human researchers with the equivalent bots, then the net load change from automation is zero, or possibly even negative.

Re: Detecting Chrome headless, the game goes on

#56

Post by jdc » Fri, Jul 19, 2019, 6:12 PM UTC

floatingatoll wrote: post #24

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Why privilege traffic based on its source (whether it's from a human or Selenium)? If some resources are expensive to serve, you can rate limit them.

Re: Detecting Chrome headless, the game goes on

#57

Post by floatingatoll » Fri, Jul 19, 2019, 6:23 PM UTC

ijpoijpoihpiuoh wrote: post #54

floatingatoll wrote: post #42

Earlier quoted context omitted.

If such a national law were passed with funding guaranteed for open publication of records, I would endorse your point of view. No such funding exists, and municipalities are regularly denied tax increases by their voters for any reason — much less public records publication that would often embarrass and humiliate those same voters. So in essence you're asking them to cut public services and staffing in order to giv…

Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access?

The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns.

Not one for-profit business does so. Why is their immoral action acceptable, when the same action by a municipality is not?

Re: Detecting Chrome headless, the game goes on

#58

Post by floatingatoll » Fri, Jul 19, 2019, 6:27 PM UTC

Pinbenterjamin wrote: post #44

floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

I don't empathize with your viewpoint because, whether it's a web scraper, or a person, the work is exactly the same. There's no additional volume, or extra steps. We just emulate a worker. We measure the value in FTEs, and when a researcher quits, we do not replace them if the appropriate FTEs have been reached with projects. It's a major benefit to the business not only because we don't have to pay another employee…

Sadly, you are correct to have realized that many posters on HN are so naive that they will offer you $0/hour consulting for your for-profit business. Posting on the HN forums means you "don't have to pay another employee" that's an expert in the field. I can't do much to prevent this, but I don't much respect it, either.

Re: Detecting Chrome headless, the game goes on

#59

Post by jsnell » Fri, Jul 19, 2019, 6:28 PM UTC

nprateem wrote: post #46

Well the user agent of chrome headless contains 'HeadlessChrome' according to this site [1]. Sure enough when I spoof my user agent to the first in the list it magically determines I'm using headless Chrome. He basically says he's inspecting user agents: > Under the hood, I only verify if browsers pretending to be Chromium-based are who they pretend to be. Thus, if your Chrome headless pretends to be Safari, I won’t…

If a browser claims to be Headless Chrome, you believe it. Nobody has a reason to lie about that. The interesting question is the opposite case: is somebody claiming to be a normal Chrome, but is actually Headless Chrome (or an automated member of some other browser family, or not a browser at all but e.g. a Python script).

So if you take a Headless Chrome instance but change the User-Agent to match that of a normal Chrome, does the detector think it's not headless?

Re: Detecting Chrome headless, the game goes on

#60

Post by dnautics » Fri, Jul 19, 2019, 6:32 PM UTC

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

putting my 'bad guy' hat on, I would think about automating via sikuli script if you had to (but only if you had to).