Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

81–90 of 143 posts

Re: Detecting Chrome headless, the game goes on

#81

Post by reaperducer »

floatingatoll wrote: post #57
ijpoijpoihpiuoh wrote: post #54

Earlier quoted context omitted.

Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does…

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access?

Currently? Not off the top of my head. But there was one that scraped municipal records in a large midwest city and made them public for free because they were confusing to get to otherwise.

Unfortunately, the company was bought by a larger company and that portion of what they did was shut down.

Re: Detecting Chrome headless, the game goes on

#82

Post by crispyporkbites »

jdc wrote: post #56
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Why privilege traffic based on its source (whether it's from a human or Selenium)? If some resources are expensive to serve, you can rate limit them.

Because some information is more valuable than the sum of its parts.

Re: Detecting Chrome headless, the game goes on

#84

Post by Pinbenterjamin »

colechristensen wrote: post #73
Pinbenterjamin wrote: post #51

Earlier quoted context omitted.

I don't have a perspective on the ethics of easier background checks. We run employment checks, the ultimate decision of whether to hire falls to the customer ALWAYS. I've seen plenty of former criminals get hired. It's a workplace culture 'thing'. The right to be forgotten is alive and well most of the time, 90% of our clients don't observe information further back than a few years. I feel like that is a fair assess…

"Just" providing the data doesn't absolve you of responsibility for the decisions others make using it. There is a point where data collection becomes unethical, and making everything fine as long as it isn't legal makes for a shitty society. (i.e. legislating behavior should be a last resort not a first judgement on right and wrong) I don't know precisely where that point is, but automated scraping of social media p…

I still don't agree. The whole reason this business exists is to remove the cost from all the industries that need to run background checks.

I think the extent and reason for the checks aren't apparent. So I'll give a few examples where we have high volume and I hope that will enlighten you as to the reason why there are so many players in the industry.

The highest volume checks are around the medical and teaching fields. We often run 6-month, to one year recurring checks on teachers and doctors to ensure licenses and certifications are still active. As well as necessary immunizations to work in their environments.

Do you expect a low margin industry like teaching to staff a full time employee to do nothing but run background checks? They want them done and the schools have access to the information, it's just much easier for them to pay us a few dollars an employee and get a nice report than do the legwork themselves.

Additionally, incurring the cost of access for the relevant data is a barrier for companies without a bunch of cash laying around.

We don't solicit companies with incriminating information about their employees, it's a necessary part to a safe environment.

Re: Detecting Chrome headless, the game goes on

#85

Post by Pinbenterjamin »

turtlebits wrote: post #19
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

The right thing to do would be to reach out to those sites and see if they is they have paid options for getting the data you need.

Well, when that option is available, as in the case of something like SAMBA WEB MVR, we absolutely opt for that instead, and pay our dues.

Re: Detecting Chrome headless, the game goes on

#86

Post by Pinbenterjamin »

curun1r wrote: post #66
Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

> Where do I start? Study the chromium source? I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user. Where do you run your scraping operation? Is it in AWS or some other hosting provider, beca…

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs.

We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site.

Some sites are just so damn good and detecting us and I just don't get it.

Re: Detecting Chrome headless, the game goes on

#87

Post by AznHisoka »

driverdan wrote: post #78
AznHisoka wrote: post #61

Earlier quoted context omitted.

Because of the issues the article described: detection of headless crawlers/bots/etc

You can automate a regular browser. It doesn't have to be headless.

Unfortunately, there are some sites that can even detect regular automated browser sessions.

Re: Detecting Chrome headless, the game goes on

#88

Post by fjp »

I work in telecom and we interface with large carriers like AT&T, Verizon, etc. We use headless browsers to automate processes using their 15-year-old admin portals, since the carriers simply refuse to provide an API, or one that works acceptably.

Thankfully they're also so technologically slow that they never change the websites or do any kind of headless detection. Its works, and allows us to offer automated [process] to our customers, but it seems so fragile. Just give us a damn API.

Re: Detecting Chrome headless, the game goes on

#89

Post by bsmith0 »

jadell wrote: post #20
mcescalante wrote: post #10

Earlier quoted context omitted.

I'm not sure if diving deep into the page will yield results of how it's done. The page's javascript does a POST to a backend with the browser's fingerprint, and the server does all the "magic" where we can't see it. Unless there is new fingerprint info that is being sent to the server that wasn't around before, I'm skeptical about the javascript in the page revealing the full technique.

The "You are/are not" message seems to be included in the page source before any Javascript runs. Is it possible there are detectable differences in the original HTTP request itself?

Yep, you got it, checkout the top commment on this thread

Re: Detecting Chrome headless, the game goes on

#90

Post by pyrale »

stickfigure wrote: post #76
floatingatoll wrote: post #57

Earlier quoted context omitted.

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does…

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking…

It is not obvious to me that reducing the cost to consult public data is necessarily a good thing. Just because this data is accessible, it should not amways also be accessible inexpensively. Example given: trial records should be public but it would probably not be nice to have all your judicial record displayed in people's glasses.
Post reply on HN