Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Detecting Chrome headless, the game goes on

#41

Post by huhtenberg »

stickfigure wrote: post #38
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form. In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

Public records are published based on certain demand assumptions.

If a real-world demand for, say, some GIS data is hundreds of requests per day, then a crawler that comes in with hundreds requests PER MINUTE will obviously stress the infrastructure. Adjusting infrastructure to cope is not an instant process, nor is it a sure thing to begin with given all the budgeting formalities. So your "civic duty" will ultimately result in destruction of these services, because they simply don't have the means to deal with such thoughtless activism.

Re: Detecting Chrome headless, the game goes on

#42

Post by floatingatoll »

stickfigure wrote: post #38
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form. In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

If such a national law were passed with funding guaranteed for open publication of records, I would endorse your point of view.

No such funding exists, and municipalities are regularly denied tax increases by their voters for any reason — much less public records publication that would often embarrass and humiliate those same voters.

So in essence you're asking them to cut public services and staffing in order to give hundreds of dollars of IT costs a month to for-profit businesses who can't be bothered to pay some small fraction of their revenue for the costs of delivering those records.

It is our civic responsibility to republish those records for free as citizens. Doing so for profit at the expense of citizens is unethical.

If OP republishes all records received in a freely-downloadable, unrestricted form, then I would happily help them fix their scrapers. They, of course, do not.

Re: Detecting Chrome headless, the game goes on

#43

Post by hannasanarion »

turtlebits wrote: post #19
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

The right thing to do would be to reach out to those sites and see if they is they have paid options for getting the data you need.

And what happens when they ignore you? I've reached out to tons of website operators to ask for machine readable access to their data on academic, personal and professional projects, I have never gotten a reply and had to resort to scraping.

Re: Detecting Chrome headless, the game goes on

#44

Post by Pinbenterjamin »

floatingatoll wrote: post #24
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

I don't empathize with your viewpoint because, whether it's a web scraper, or a person, the work is exactly the same. There's no additional volume, or extra steps. We just emulate a worker.

We measure the value in FTEs, and when a researcher quits, we do not replace them if the appropriate FTEs have been reached with projects.

It's a major benefit to the business not only because we don't have to pay another employee, but we can reduce training costs, and costs incurred by mistakes. We can also adjust execution of one of these agents, which normally would require rearrangement of work instructions, and retraining.

These are public records, 90% of them do not have integrations for automated systems, and those that do, we utilize. They are typically search boxes with results. We are not circumventing any type of cost that would otherwise be incurred.

We do not log any of the results, store them locally, or maintain any of the PII with each search. If a case was searched 20 minutes ago, and comes up again, we rerun the entire thing just as a human would.

Finally, to your point about 'help me with my homework', I consider posting on the HN forums homework for this type of research. There are a diverse set of talented developers on here with esoteric experience. The fact that an article related to the work I do came up on here, I thought, was an excellent opportunity to seek advice and perspective.

Re: Detecting Chrome headless, the game goes on

#45

Post by paganel »

elorant wrote: post #40
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Have you tried loading a full browser session? Not just headless.

Not the OP, but I did that about 12 years ago, with Firefox. My boss at the time had asked me to parse some public institution website that was quite difficult to write a parser for directly in Python, so in the end we just decided to write a quick extension for Firefox and let an instance of it run on a spare computer. That public institution website had some JS bug that would cause FF to gobble up memory pretty fast, but we also solved that by automatically restarting FF at certain intervals (or when we noticed something was off).

Not sure if people do this sort of things nowadays.

Re: Detecting Chrome headless, the game goes on

#46

Post by nprateem »

Well the user agent of chrome headless contains 'HeadlessChrome' according to this site [1]. Sure enough when I spoof my user agent to the first in the list it magically determines I'm using headless Chrome.

He basically says he's inspecting user agents:

> Under the hood, I only verify if browsers pretending to be Chromium-based are who they pretend to be. Thus, if your Chrome headless pretends to be Safari, I won’t catch it with my technique.

Maybe I should apply for a PhD too.

[1] https://user-agents.net/browsers/headless-chrome

Re: Detecting Chrome headless, the game goes on

#47

Post by Pinbenterjamin »

ProCicero wrote: post #31
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I'm very curious to know how you are able to get precise and accurate enough identification information from public websites to be able to credibly run a "background check" on someone. I used to work in the criminal justice system, and had unlimited access to every single criminal case initiated in my state going back for almost 40 years. It's difficult enough for a trained person to do it by hand, let alone automati…

There are a number of ways we do this.

First, the process of automating a source is not as simple as 'grab data, send to person that creates the case'.

We have many many layers of precaution and validation both by humans and other automated systems, that helps guarantee accuracy.

On top of this, even public records has reporting rules in the industry. There are dates, specific charges, charge types (Misdemeanor/Felony), and a battery of other rules that the information is processed through in order to ensure we do not report information that we are not allowed.

We always lean to the side of throwing a case to a human. In the circumstance that anything new, unrecognized, or even slightly off happens, we toss the case to a team that processes the information by hand. At that point, we are simply a scraper for information and we cut out the step of having a human order and retrieve results.

We do not go back 40 years. Industry standards dictate that most records older than 7 years are expunged from Employment background checks. And most of our clients don't care about more than 3 years worth, with exceptions like Murder, Federal Crimes, and some obviously heinous things.

We also run a number of other tests, outside of public records to provide full background data. We have integrations with major labs to schedule drug screens, we allow those who are having a background check run on them to fill out an application to provide reasoning and information from their point of view to allow customers to empathize with an employee.

We also have a robust dispute system. The person having a background check run on them receives the report before the client requesting it in order to review the results and dispute anything they find wrong. These cases are always handled by a human, and often involve intensive research, no cost spared, to ensure the accuracy of the report.

There's a plethora of other things I'm missing, but if you have any specific questions, I'm happy to answer.

*EDIT

To clarify, there is a lot of information in public records. It isn't unclear or ambiguous at all. Motor Vehicle and Court records are extremely in-depth and spare no detail.

Re: Detecting Chrome headless, the game goes on

#48

Post by slaymaker1907 »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I personally think that it maybe be ethically questionable to be making background checks easier. There is a reason why the right to be forgotten is becoming a thing in various jurisdictions and lack of easy access for sensitive data is one countermeasure to try and counterbalance the need for public access to data with the right to privacy for individuals.

Re: Detecting Chrome headless, the game goes on

#50

Post by jadell »

It seems he's doing something with header detection. I used Puppeteer to play around with the site and various configurations I use when scraping.

In headless Chrome, the "Accept-Language" header is not sent. In Puppeteer, one can force the header to be sent by doing:

  page.setExtraHTTPHeaders({ 'Accept-Language': 'en-US,en;q=0.9' })
However, Puppeteer sends that header as lowercase:

  accept-language: en-US,en;q=0.9
So it seems the detection is as simply: if there is no 'Accept-Language' header (case-sensitive), then "Headless Chrome"; else, "Not Headless Chrome".

This is a completely server-side check, which is why he can say the fpcollect client-side javascript library isn't involved.

Here are some curl commands that demonstrate:

Detected: not headless

  curl 'https://arh.antoinevastel.com/bots/areyouheadless' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36' \
  -H 'Accept-Language: en-US,en;q=0.9'
Detected: headless

  curl 'https://arh.antoinevastel.com/bots/areyouheadless' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36'
Detected: headless

  curl 'https://arh.antoinevastel.com/bots/areyouheadless' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36' \
  -H 'accept-language: en-US,en;q=0.9'
Post reply on HN