Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

71–80 of 143 posts

Re: Detecting Chrome headless, the game goes on

#71

Post by lyxsus1 »

jadell wrote: post #25
ryandrake wrote: post #21

Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Intern…

I agonize about this every day, since a large part of my job is aggregating data from many sites that seem hell-bent on not letting anyone access it without going one-form-at-a-time through their crap UI. The thing is, we would gladly pay these companies for an API or even just a periodic data-dump of what we need. We've even offered to some of them to write and maintain the API for them. They're not interested, for…

Sometimes when I'm thinking about it and what 95% of developers are working on, it feels like a planet-wide charity project against unemployment.

Re: Detecting Chrome headless, the game goes on

#72

Post by jadell »

jadell wrote: post #50

It seems he's doing something with header detection. I used Puppeteer to play around with the site and various configurations I use when scraping. In headless Chrome, the "Accept-Language" header is not sent. In Puppeteer, one can force the header to be sent by doing: page.setExtraHTTPHeaders({ 'Accept-Language': 'en-US,en;q=0.9' }) However, Puppeteer sends that header as lowercase: accept-language: en-US,en;q=0.9 So…

As a followup, if you have the ability to modify the Chrome/Chromium command line arguments, using the following option completely fools the detection:

  --lang=en-US,en;q=0.9
You can prove this with the following Puppeteer script:

  (async () => {
      const puppeteer = require('puppeteer');
      const browserOpts = {
          headless: true,
          args: [
              '--no-sandbox',
              '--disable-setuid-sandbox',
              '--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3803.0 Safari/537.36',
              // THIS IS THE KEY BIT!
              '--lang=en-US,en;q=0.9',
          ],
      };
  
      const browser = await puppeteer.launch(browserOpts);
      const page = await browser.newPage();
      await page.goto('https://arh.antoinevastel.com/bots/areyouheadless');
      await page.screenshot({ path: 'areyouheadless.png' });
      await browser.close();
  })();

Re: Detecting Chrome headless, the game goes on

#73

Post by colechristensen »

Pinbenterjamin wrote: post #51
slaymaker1907 wrote: post #48

Earlier quoted context omitted.

I personally think that it maybe be ethically questionable to be making background checks easier. There is a reason why the right to be forgotten is becoming a thing in various jurisdictions and lack of easy access for sensitive data is one countermeasure to try and counterbalance the need for public access to data with the right to privacy for individuals.

I don't have a perspective on the ethics of easier background checks. We run employment checks, the ultimate decision of whether to hire falls to the customer ALWAYS. I've seen plenty of former criminals get hired. It's a workplace culture 'thing'. The right to be forgotten is alive and well most of the time, 90% of our clients don't observe information further back than a few years. I feel like that is a fair assess…

"Just" providing the data doesn't absolve you of responsibility for the decisions others make using it.

There is a point where data collection becomes unethical, and making everything fine as long as it isn't legal makes for a shitty society. (i.e. legislating behavior should be a last resort not a first judgement on right and wrong)

I don't know precisely where that point is, but automated scraping of social media probably is past (automated scraping of judicial records? probably ok)

Re: Detecting Chrome headless, the game goes on

#74

Post by vageli »

samcal wrote: post #32
satyrnein wrote: post #29

Earlier quoted context omitted.

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.

> Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.

Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to every site they wanted to crawl?

How many companies out there complain about being scraped by Google? How many companies benefit from search-driven traffic?

Re: Detecting Chrome headless, the game goes on

#75

Post by danShumway »

floatingatoll wrote: post #58
Pinbenterjamin wrote: post #44

Earlier quoted context omitted.

I don't empathize with your viewpoint because, whether it's a web scraper, or a person, the work is exactly the same. There's no additional volume, or extra steps. We just emulate a worker. We measure the value in FTEs, and when a researcher quits, we do not replace them if the appropriate FTEs have been reached with projects. It's a major benefit to the business not only because we don't have to pay another employee…

Sadly, you are correct to have realized that many posters on HN are so naive that they will offer you $0/hour consulting for your for-profit business. Posting on the HN forums means you "don't have to pay another employee" that's an expert in the field. I can't do much to prevent this, but I don't much respect it, either.

I'm glad that sites like Wikipedia, StackOverflow, and HN exist. I don't think the world is a worse place because they exist, and I respect the people who post there.

This is the same attitude that says, "why would someone just give away Open Source software when they could build a SaaS business instead?"

Re: Detecting Chrome headless, the game goes on

#76

Post by stickfigure »

floatingatoll wrote: post #57
ijpoijpoihpiuoh wrote: post #54

Earlier quoted context omitted.

Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.

Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does…

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries.

The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines.

It's obvious how we got to this point (before the internet, you obtained public records by walking into an office) but it's long past time to change. We don't need fancy web forms to search and find data; cut all that out and just provide data in machine readable form to anyone who wants it.

Someone will build a pretty commercial interface to public records data. Chances are, they can do it for less than the 8-figure sum required for UI development in the public sector. Win-win.

Re: Detecting Chrome headless, the game goes on

#77

Post by eastendguy »

All this can be avoided (from a scraper's perspective) by using the Selenium IDE++ project. It adds a command line to Chrome and Firefox to run scripts. See https://ui.vision/docs#cmd and https://ui.vision/docs/selenium-ide/web-scraping

=> Using Chrome directly is slower, but undetectable.

Re: Detecting Chrome headless, the game goes on

#78

Post by driverdan »

AznHisoka wrote: post #61
TomMarius wrote: post #35

Earlier quoted context omitted.

Why would you do that when you can automate it?

Because of the issues the article described: detection of headless crawlers/bots/etc

You can automate a regular browser. It doesn't have to be headless.

Re: Detecting Chrome headless, the game goes on

#79

Post by driverdan »

jadell wrote: post #25
ryandrake wrote: post #21

Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Intern…

I agonize about this every day, since a large part of my job is aggregating data from many sites that seem hell-bent on not letting anyone access it without going one-form-at-a-time through their crap UI. The thing is, we would gladly pay these companies for an API or even just a periodic data-dump of what we need. We've even offered to some of them to write and maintain the API for them. They're not interested, for…

The travel industry is highly protective of its data. It's my understanding that they consider it proprietary and only sell it to those who they deem worthy.

Re: Detecting Chrome headless, the game goes on

#80

Post by greglindahl »

klenwell wrote: post #36
satyrnein wrote: post #29

Earlier quoted context omitted.

Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach. If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!

I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.

That's not true in the actual web, however.

The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting.

Post reply on HN