Detecting Chrome headless, the game goes on • Hacker News Board

Re: Detecting Chrome headless, the game goes on

#91

Post by z3t4 » Fri, Jul 19, 2019, 9:25 PM UTC

stickfigure wrote: post #76

floatingatoll wrote: post #57

Earlier quoted context omitted.

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does…

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking…

Some "public" records are in the gray area as in; should or should they not (black and white) be published. For example salaries, the employer might forbid disclosing salaries, but anyone can just request anyone's salary from the government because its public. But if they could be downloaded from an FTP ...

Re: Detecting Chrome headless, the game goes on

#92

Post by brlewis » Fri, Jul 19, 2019, 9:33 PM UTC

curun1r wrote: post #66

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

> Where do I start? Study the chromium source? I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user. Where do you run your scraping operation? Is it in AWS or some other hosting provider, beca…

> rate limit, including adding random jitter to mimic the way a human might use a browser

Even if you aren't trying to disguise anything, adding some randomness helps avoid one particular bad pattern with operations on a network. I recall the pattern being called "network synchronization" but I can't get good search results for that.

Re: Detecting Chrome headless, the game goes on

#93

Post by floatingatoll » Fri, Jul 19, 2019, 9:34 PM UTC

danShumway wrote: post #75

floatingatoll wrote: post #58

Earlier quoted context omitted.

Sadly, you are correct to have realized that many posters on HN are so naive that they will offer you $0/hour consulting for your for-profit business. Posting on the HN forums means you "don't have to pay another employee" that's an expert in the field. I can't do much to prevent this, but I don't much respect it, either.

I'm glad that sites like Wikipedia, StackOverflow, and HN exist. I don't think the world is a worse place because they exist, and I respect the people who post there. This is the same attitude that says, "why would someone just give away Open Source software when they could build a SaaS business instead?"

I don’t think Stackoverflow for “how can I avoid paying a municipality a reasonable public records fee” should exist, but I do endorse Stackoverflow in general. You’ll have to do what you will with that; generalizing my point to “all Stackoverflow” is certainly wrong, though.

Re: Detecting Chrome headless, the game goes on

#94

Post by floatingatoll » Fri, Jul 19, 2019, 9:38 PM UTC

satyrnein wrote: post #29

floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

We found at one job that approximate one quarter of well-known search engines blatantly use robots.txt noindex declarations as a list of URLs to index, and one openly mocked us for asking them to stop.

Voluntary honor systems don’t work, because there’s no way to compel non-compliers to stop other than standard “anti-attacker arms race” approaches, such as the obstacle described at the head of this thread.

Re: Detecting Chrome headless, the game goes on

#95

Post by thekyle » Fri, Jul 19, 2019, 9:42 PM UTC

z3t4 wrote: post #91

stickfigure wrote: post #76

Earlier quoted context omitted.

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking…

Some "public" records are in the gray area as in; should or should they not (black and white) be published. For example salaries, the employer might forbid disclosing salaries, but anyone can just request anyone's salary from the government because its public. But if they could be downloaded from an FTP ...

What government agency allows you to see arbitrary other peoples salaries?

Re: Detecting Chrome headless, the game goes on

#96

Post by brlewis » Fri, Jul 19, 2019, 9:45 PM UTC

ilikehurdles wrote: post #37

ProCicero wrote: post #31

Earlier quoted context omitted.

I'm very curious to know how you are able to get precise and accurate enough identification information from public websites to be able to credibly run a "background check" on someone. I used to work in the criminal justice system, and had unlimited access to every single criminal case initiated in my state going back for almost 40 years. It's difficult enough for a trained person to do it by hand, let alone automati…

My wife's SSN/credit history/online identities have in the past been mistakenly tied up with her sibling's. This has since been corrected with all the appropriate agencies and organizations. However, from what I've noticed of search results over time, these background check (AND identity verification) sites crawl each other and create a kind of feedback loop, as I've been noticing that some of these pages will falsel…

> These services cause more problems than they solve, and should require consent, oversight, and civil or criminal penalties associated with a failure to meet high quality standards.

Existing law does not proscribe recklessly sharing damaging false information about people?

Re: Detecting Chrome headless, the game goes on

#97

Post by nutjob2 » Fri, Jul 19, 2019, 10:01 PM UTC

Pinbenterjamin wrote: post #86

curun1r wrote: post #66

Earlier quoted context omitted.

> Where do I start? Study the chromium source? I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user. Where do you run your scraping operation? Is it in AWS or some other hosting provider, beca…

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

They can characterise the (browsing) behaviour of all their visitors, and then further characterise those who fall outside their "normal" thresholds. The outsiders that exhibit some sort of correlation (ie their characteristics are not independent of each other) are banned. Any quirks or patterns your systems have would be identifiable as "artificial", and even those that are randomised or seek to emulate humans will have features that are identifiable. An NDA is ineffective against machine learning.

The countermeasure would be to have a bunch of humans use the websites in any way they want, totally undirected, then use the totality of that browsing to facilitate your scraping probabilistically. It would be less efficient, but very difficult to catch.

Re: Detecting Chrome headless, the game goes on

#98

Post by foob » Fri, Jul 19, 2019, 10:06 PM UTC

I'm the other half of the cat and mouse game that Antoine is referring to, and I just wrote another rebuttal that people here might find interesting [1]. It goes into a little more detail about what his test site is actually doing, and also walks through the process of writing a Puppeteer script to bypass the tests.

- [1] -https://www.tenantbase.com/tech/blog/cat-and-mouse/

Re: Detecting Chrome headless, the game goes on

#99

Post by cameronbrown » Fri, Jul 19, 2019, 10:17 PM UTC

pyrale wrote: post #90

stickfigure wrote: post #76

Earlier quoted context omitted.

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking…

It is not obvious to me that reducing the cost to consult public data is necessarily a good thing. Just because this data is accessible, it should not amways also be accessible inexpensively. Example given: trial records should be public but it would probably not be nice to have all your judicial record displayed in people's glasses.

Disagree. It's inherently in the public interest to have access to this data as easily as possible. If it's too embarrassing then that's a cultural problem.

Re: Detecting Chrome headless, the game goes on

#100

Post by colechristensen » Fri, Jul 19, 2019, 10:32 PM UTC

Pinbenterjamin wrote: post #84

colechristensen wrote: post #73

Earlier quoted context omitted.

"Just" providing the data doesn't absolve you of responsibility for the decisions others make using it. There is a point where data collection becomes unethical, and making everything fine as long as it isn't legal makes for a shitty society. (i.e. legislating behavior should be a last resort not a first judgement on right and wrong) I don't know precisely where that point is, but automated scraping of social media p…

I still don't agree. The whole reason this business exists is to remove the cost from all the industries that need to run background checks. I think the extent and reason for the checks aren't apparent. So I'll give a few examples where we have high volume and I hope that will enlighten you as to the reason why there are so many players in the industry. The highest volume checks are around the medical and teaching fi…

shrug not trying to imply it is all bad. The nature of the information is important. Professional certification or licensing checks are obviously harmless.

What isn't harmless is gathering information about the private lives of people (even when done in the public eye) in ways that are difficult, labor intensive, or impossible without automation.