Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting Chrome headless, the game goes on

antoinevastel.com

Post reply on HN
Print view View original

111–120 of 143 posts

Re: Detecting Chrome headless, the game goes on

#111

Post by pault »

UweSchmidt wrote: post #109
Pinbenterjamin wrote: post #105

Earlier quoted context omitted.

That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.

In the grand scheme of things all of this is a wasteful process. Maybe you could direct your worklife towards other challenges that are more rewarding for society and equally profitable?

I think that's unjustified and a little rude. OP is providing an automated service for publicly accessible data that isn't accessible for automation. If the sources are notified and they are operating within the confines of the law, this is no different than writing a search engine crawler.

Re: Detecting Chrome headless, the game goes on

#112

Post by BEEdwards »

z3t4 wrote: post #91
stickfigure wrote: post #76

Earlier quoted context omitted.

There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking…

Some "public" records are in the gray area as in; should or should they not (black and white) be published. For example salaries, the employer might forbid disclosing salaries, but anyone can just request anyone's salary from the government because its public. But if they could be downloaded from an FTP ...

In the US it is illegal for employers to forbid disclosing salaries.

Discussing salaries is a taboo created by industries to stifle wages.

https://www.monster.com/career-advice/article/truth-about-di...

Re: Detecting Chrome headless, the game goes on

#113

Post by floatingatoll »

jdc wrote: post #107
floatingatoll wrote: post #94

Earlier quoted context omitted.

We found at one job that approximate one quarter of well-known search engines blatantly use robots.txt noindex declarations as a list of URLs to index, and one openly mocked us for asking them to stop. Voluntary honor systems don’t work, because there’s no way to compel non-compliers to stop other than standard “anti-attacker arms race” approaches, such as the obstacle described at the head of this thread.

It sounds like scraping is a big problem for you guys. What kind of outfit is it, if you don't mind me asking?

Drop me an email and I’m happy to describe further.

Re: Detecting Chrome headless, the game goes on

#114

Post by pault »

Pinbenterjamin wrote: post #44
floatingatoll wrote: post #24

Earlier quoted context omitted.

Reducing your business costs by scraping a public access website is often considered an alternative to paying the business costs of the website operator. Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator? If you're costing them money to reduce your own bottom line without their express written consent, that makes yo…

I don't empathize with your viewpoint because, whether it's a web scraper, or a person, the work is exactly the same. There's no additional volume, or extra steps. We just emulate a worker. We measure the value in FTEs, and when a researcher quits, we do not replace them if the appropriate FTEs have been reached with projects. It's a major benefit to the business not only because we don't have to pay another employee…

Don't be discouraged by the spiteful kneejerk reactions in this thread. HN is a diverse place and some commenters get triggered by an association with one of their pet peeves and launch into a rant without taking time to assess the nuance of your position. I've been the butt of this behavior a few times and it can be pretty toxic.

Re: Detecting Chrome headless, the game goes on

#115

Post by pault »

paganel wrote: post #45
elorant wrote: post #40

Earlier quoted context omitted.

Have you tried loading a full browser session? Not just headless.

Not the OP, but I did that about 12 years ago, with Firefox. My boss at the time had asked me to parse some public institution website that was quite difficult to write a parser for directly in Python, so in the end we just decided to write a quick extension for Firefox and let an instance of it run on a spare computer. That public institution website had some JS bug that would cause FF to gobble up memory pretty fas…

When I'm doing personal scraping, I just write a chrome extension. You can find boilerplates that are super easy to set up, and they persist in a background thread between page loads. It's really easy to collect the data and log it in the console or send it to a local API or database. It's the lowest effort method of scraping I know, and you can monitor it while it runs to make sure it doesn't get hung up on some edge case.

Re: Detecting Chrome headless, the game goes on

#116

Post by morpheuskafka »

underwater wrote: post #101
Pinbenterjamin wrote: post #86

Earlier quoted context omitted.

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good…

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

Although his charges were ridiculous, they involved physically connecting to a secure network without permission, not just scraping the public part of pages from his own networks.

Re: Detecting Chrome headless, the game goes on

#117

Post by pault »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

Have you tried writing a chrome extension and running it in a desktop browser instance? It's super easy to set up and shouldn't appear any different than a regular user if you rate limit and add some randomness to the input events.

Re: Detecting Chrome headless, the game goes on

#118

Post by formercoder »

UweSchmidt wrote: post #109
Pinbenterjamin wrote: post #105

Earlier quoted context omitted.

That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.

In the grand scheme of things all of this is a wasteful process. Maybe you could direct your worklife towards other challenges that are more rewarding for society and equally profitable?

OP is being reasonably compensated for something that is perfectly legal.

Re: Detecting Chrome headless, the game goes on

#119

Post by underwater »

Havoc wrote: post #106
underwater wrote: post #101

Earlier quoted context omitted.

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

>Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Don't think connecting a computer to a private network to suck up subscriber data is comparable to scraping publicly accessible internet content.

My understanding of the case is that he was charged with evading JSTOR security, not for accessing the MIT network.

Re: Detecting Chrome headless, the game goes on

#120

Post by katzgrau »

Pinbenterjamin wrote: post #16

I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers. Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless…

I run an ad delivery platform (hey, we're both popular) and I detect and block bots because they tend to inadvertly drive up engagement counts on ad campaigns, creating a situation where publishers can't be confident in their numbers. Some clients have their own tech to do the the same.
Post reply on HN