Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

31–40 of 143 posts

Re: Detecting Chrome headless, the game goes on

#31

Post by ProCicero »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I'm very curious to know how you are able to get precise and accurate enough identification information from public websites to be able to credibly run a "background check" on someone. I used to work in the criminal justice system, and had unlimited access to every single criminal case initiated in my state going back for almost 40 years. It's difficult enough for a trained person to do it by hand, let alone automating it. How do you provide any guarantee of accuracy?

Re: Detecting Chrome headless, the game goes on

#32

Post by samcal »

satyrnein wrote: post #29
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.

Re: Detecting Chrome headless, the game goes on

#33

Post by heyoni »

ProCicero wrote: post #31
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I'm very curious to know how you are able to get precise and accurate enough identification information from public websites to be able to credibly run a "background check" on someone. I used to work in the criminal justice system, and had unlimited access to every single criminal case initiated in my state going back for almost 40 years. It's difficult enough for a trained person to do it by hand, let alone automati…

I would imagine a live person audits the information collected by the scrapers, thereby eliminating the hassle of collecting it from multiple different sources.

As a private person, we only have access to court documents on a state or county base. Any central database we have access to would be made my scrapers.

Re: Detecting Chrome headless, the game goes on

#34

Post by AznHisoka »

I think there might be a market for "human crawlers". Just like people use Mechanical Turk to get humans to beat CAPTCHAs, you could use it to get humans to visit a web page for you, and return its HTML source. There are of course residential proxy services (ie HolaVPN), but they're still technically can be detected.

Re: Detecting Chrome headless, the game goes on

#35

Post by TomMarius »

AznHisoka wrote: post #34

I think there might be a market for "human crawlers". Just like people use Mechanical Turk to get humans to beat CAPTCHAs, you could use it to get humans to visit a web page for you, and return its HTML source. There are of course residential proxy services (ie HolaVPN), but they're still technically can be detected.

Why would you do that when you can automate it?

Re: Detecting Chrome headless, the game goes on

#36

Post by klenwell »

satyrnein wrote: post #29
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.

Re: Detecting Chrome headless, the game goes on

#37

Post by ilikehurdles »

ProCicero wrote: post #31
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I'm very curious to know how you are able to get precise and accurate enough identification information from public websites to be able to credibly run a "background check" on someone. I used to work in the criminal justice system, and had unlimited access to every single criminal case initiated in my state going back for almost 40 years. It's difficult enough for a trained person to do it by hand, let alone automati…

My wife's SSN/credit history/online identities have in the past been mistakenly tied up with her sibling's. This has since been corrected with all the appropriate agencies and organizations.

However, from what I've noticed of search results over time, these background check (AND identity verification) sites crawl each other and create a kind of feedback loop, as I've been noticing that some of these pages will falsely report parts of her sibling's background among her own, and falsely flag her as having certain ugly events in her past that don't actually belong to her. This is concerning, as her career area cares a lot about employees having a clean background, and employers using these cheap automated options see cheap, inaccurate results. She has a squeaky clean background with a high credit score and impressive educational credentials, while her sibling has had run-ins with the law and bad debts. I'm concerned about how this will affect her future career prospects.

Beyond background checks, identity verification is a big concern as well. You may have noticed some services ask you to confirm certain facts about your past (street names of where you've lived, schools you attended, jobs and cars you've held). When pulling her credit bureau reports, some of these verifications required confirming facts about her sibling rather than her own in order to gain access.

Like I said, these issues have been fixed with all the "official" record-keeping organizations; however, since the fix, I've been noticing increasing issues with the original mistakes propagating to 3rd-party background-check organizations.

These services cause more problems than they solve, and should require consent, oversight, and civil or criminal penalties associated with a failure to meet high quality standards.

Re: Detecting Chrome headless, the game goes on

#38

Post by stickfigure »

floatingatoll wrote: post #24
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Public records are public.

The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form.

In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

Re: Detecting Chrome headless, the game goes on

#39

Post by huhtenberg »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

If you don't inspect and respect robots.txt, you shouldn't be surprised by sites actively blocking your crawlers. Ditto for when you try and work around crawling restrictions by hiding behind real browser UAs.

Re: Detecting Chrome headless, the game goes on

#40

Post by elorant »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Have you tried loading a full browser session? Not just headless.
Post reply on HN