Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Internet. If your data is not your competitive advantage, then why bother wasting all this development effort stopping browsers from browsing it? So much waste on both sides.
Detecting Chrome headless, the game goes on
21–30 of 143 posts
Re: Detecting Chrome headless, the game goes on
#22There are additional tests included in https://arh.antoinevastel.com/javascripts/fpCollect.min.js that do not exist in the GitHub repository over at https://github.com/antoinevastel/fp-collect . redPill: function() { for (var e = performance.now(), n = 0, t = 0, r = [], o = performance.now(); o - e
So it's just measuring computation speed of math calculations? That doesn't sound very reliable.
Re: Detecting Chrome headless, the game goes on
#23Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Intern…
who says this is zero sum? in the limit, it seems like a loose/loose situation to me. or at "best" modest rewards for [spammers|scrapers|...] at tremendous (spread out over the population) cost in loss of usability and compute cycles and development effort
Re: Detecting Chrome headless, the game goes on
#24I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator.
Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator?
If you're costing them money to reduce your own bottom line without their express written consent, that makes you "the bad guy". Offsetting costs onto an unwitting, non-consenting third party is an unethical approach to doing business.
I interpret your request as a similar problem to "help me with my homework problem". I could dig up papers and studies, but at the end of the day, you need to go do your homework. Reach out to each municipality and figure out a business arrangement with them that satisfies your needs. It's possible they do not wish you to perform this activity, in which case you will either need to violate their intent for your own profit using scraping or accede to their wishes and stop scraping their municipality. That's your homework as a for-profit business.
Re: Detecting Chrome headless, the game goes on
#25Out of all the zero-sum tech arms races (increasingly complex DRM, SPAM senders/blockers, software crackers vs. copy protection, code obfuscation) this one seems to me to be the stupidest. Here we have people putting data out in public for free, for anyone to access, and then agonizing over how someone accesses it. If some data is your company's secret sauce, your competitive advantage, don't put it out on the Intern…
I agonize about this every day, since a large part of my job is aggregating data from many sites that seem hell-bent on not letting anyone access it without going one-form-at-a-time through their crap UI.
The thing is, we would gladly pay these companies for an API or even just a periodic data-dump of what we need. We've even offered to some of them to write and maintain the API for them. They're not interested, for various industry-specific reasons.
I often wonder how much developer time and money are wasted in total between them blocking and devs working around their blocks.
Re: Detecting Chrome headless, the game goes on
#26I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
To the siblings wondering about reaching out to the sites and offering to pay for the data: I'm not parent poster, but where I work, we absolutely have reached out. We've even offered to build and maintain the systems/APIs/etc we'd need at our own expense in addition to paying for the data. None of the companies we've reached out to seem interested in providing easy access to their data.
Re: Detecting Chrome headless, the game goes on
#27I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
i run a scraper on craigs list style marketplace for my country, they have now one of those commercial scraping protection, that i trivialy escape with basicaly adding random string to a url. try how they work and then create a workaround, i think most of them use some of those comercial solutions.
i do my scraping just for myself. maybe if i would scale it up they would detect me.
Re: Detecting Chrome headless, the game goes on
#28Earlier quoted context omitted.
I'm not sure if diving deep into the page will yield results of how it's done. The page's javascript does a POST to a backend with the browser's fingerprint, and the server does all the "magic" where we can't see it. Unless there is new fingerprint info that is being sent to the server that wasn't around before, I'm skeptical about the javascript in the page revealing the full technique.
The "You are/are not" message seems to be included in the page source before any Javascript runs. Is it possible there are detectable differences in the original HTTP request itself?
My guess is he's looking at XSS mitigations or similar that aren't in headless?
If it were doing something like using CSS being non-blocking (? I don't know that it is) that's a server side detection .. but that would seem to work even against spoofing.
But he says if you spoof another Chrome-based browser (Safari) he can't tell. So he's looking first at UA?? That's weird.
Re: Detecting Chrome headless, the game goes on
#29I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…
Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach.
If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!
Re: Detecting Chrome headless, the game goes on
#30I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…
If it suits your needs, please consider using Common Crawl instead.