Detecting Chrome headless, the game goes on • Hacker News Board

Re: Detecting Chrome headless, the game goes on

#101

Post by underwater » Fri, Jul 19, 2019, 10:34 PM UTC

Pinbenterjamin wrote: post #86

curun1r wrote: post #66

Earlier quoted context omitted.

> Where do I start? Study the chromium source? I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user. Where do you run your scraping operation? Is it in AWS or some other hosting provider, beca…

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing.

Given that your run this division there is a good chance you are personally liable.

Re: Detecting Chrome headless, the game goes on

#102

Post by abcpassword » Fri, Jul 19, 2019, 10:39 PM UTC

underwater wrote: post #101

Pinbenterjamin wrote: post #86

Earlier quoted context omitted.

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

He’s definitely personally liable. I was tasked with a similar thing at work and refused. Prison time isn’t worth the paycheck.

Re: Detecting Chrome headless, the game goes on

#103

Post by desas » Fri, Jul 19, 2019, 10:41 PM UTC

thekyle wrote: post #95

z3t4 wrote: post #91

Earlier quoted context omitted.

Some "public" records are in the gray area as in; should or should they not (black and white) be published. For example salaries, the employer might forbid disclosing salaries, but anyone can just request anyone's salary from the government because its public. But if they could be downloaded from an FTP ...

What government agency allows you to see arbitrary other peoples salaries?

Tax records are published in Sweden, Finland and Norway

Re: Detecting Chrome headless, the game goes on

#104

Post by Pinbenterjamin » Fri, Jul 19, 2019, 10:51 PM UTC

underwater wrote: post #101

Pinbenterjamin wrote: post #86

Earlier quoted context omitted.

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

We have an enormous legal team that communicates constantly with end points to ensure they are aware of our scraping. And as I said in another comment, we store no results other than what is already available to anyone else using the web.

We've had this division for many many years, and before my time we paid another company to do this. There's no legal issues.

Re: Detecting Chrome headless, the game goes on

#105

Post by Pinbenterjamin » Fri, Jul 19, 2019, 10:53 PM UTC

nutjob2 wrote: post #97

Pinbenterjamin wrote: post #86

Earlier quoted context omitted.

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

They can characterise the (browsing) behaviour of all their visitors, and then further characterise those who fall outside their "normal" thresholds. The outsiders that exhibit some sort of correlation (ie their characteristics are not independent of each other) are banned. Any quirks or patterns your systems have would be identifiable as "artificial", and even those that are randomised or seek to emulate humans will…

That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.

Re: Detecting Chrome headless, the game goes on

#106

Post by Havoc » Fri, Jul 19, 2019, 10:56 PM UTC

underwater wrote: post #101

Pinbenterjamin wrote: post #86

Earlier quoted context omitted.

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

>Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing.

Don't think connecting a computer to a private network to suck up subscriber data is comparable to scraping publicly accessible internet content.

Re: Detecting Chrome headless, the game goes on

#107

Post by jdc » Fri, Jul 19, 2019, 11:08 PM UTC

floatingatoll wrote: post #94

satyrnein wrote: post #29

Earlier quoted context omitted.

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

We found at one job that approximate one quarter of well-known search engines blatantly use robots.txt noindex declarations as a list of URLs to index, and one openly mocked us for asking them to stop. Voluntary honor systems don’t work, because there’s no way to compel non-compliers to stop other than standard “anti-attacker arms race” approaches, such as the obstacle described at the head of this thread.

It sounds like scraping is a big problem for you guys. What kind of outfit is it, if you don't mind me asking?

Re: Detecting Chrome headless, the game goes on

#108

Post by gnud » Fri, Jul 19, 2019, 11:11 PM UTC

vageli wrote: post #74

samcal wrote: post #32

Earlier quoted context omitted.

Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.

> Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world. Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to ever…

The alternative would have been opt-in instead of opt-out. Everything excluded by default, except what robots.txt allows you to index.

Naturally, Google didn't want that.

Re: Detecting Chrome headless, the game goes on

#109

Post by UweSchmidt » Fri, Jul 19, 2019, 11:23 PM UTC

Pinbenterjamin wrote: post #105

nutjob2 wrote: post #97

Earlier quoted context omitted.

They can characterise the (browsing) behaviour of all their visitors, and then further characterise those who fall outside their "normal" thresholds. The outsiders that exhibit some sort of correlation (ie their characteristics are not independent of each other) are banned. Any quirks or patterns your systems have would be identifiable as "artificial", and even those that are randomised or seek to emulate humans will…

That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.

In the grand scheme of things all of this is a wasteful process. Maybe you could direct your worklife towards other challenges that are more rewarding for society and equally profitable?

Re: Detecting Chrome headless, the game goes on

#110

Post by 3xblah » Fri, Jul 19, 2019, 11:33 PM UTC

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

If you are not doing anything subversive then can you share with us some examples of the sites that are selectively blocking you? And give us an example of the public information that is sought. Perhaps disclosing any of the sites puts you at disadvantage versus potential competitors? It does not make much sense to block access to public information, assuming you are not interfering with others' access.