Ask HN: What are you using headless Chrome for? • Hacker News Board

Re: Ask HN: What are you using headless Chrome for?

#11

Post by mothinx » Thu, Sep 24, 2020, 5:41 AM UTC

I use it for a trading bot on Fifa 2020 Fut mode

Re: Ask HN: What are you using headless Chrome for?

#12

Post by jamil7 » Thu, Sep 24, 2020, 6:13 AM UTC

jmnicolas wrote: post #3

I use it to get rendered web pages I want to scrap.

I was doing the same for an article reader / archiver I built but it was slow. Did you ever figure out how to speed it up? Cancelling image and media requests helped but it increased the likelihood of being flagged as a bot.

Re: Ask HN: What are you using headless Chrome for?

#13

Post by jmnicolas » Thu, Sep 24, 2020, 8:53 AM UTC

jamil7 wrote: post #12

jmnicolas wrote: post #3

I use it to get rendered web pages I want to scrap.

I was doing the same for an article reader / archiver I built but it was slow. Did you ever figure out how to speed it up? Cancelling image and media requests helped but it increased the likelihood of being flagged as a bot.

Sorry I never looked at the perf aspect of it since my needs were modest (less than 50 web pages per day). The first thing that would come to my mind is to parallelize it but then it's going to become pretty complex.

Re: Ask HN: What are you using headless Chrome for?

#14

Post by joefarish » Thu, Sep 24, 2020, 11:58 AM UTC

mothinx wrote: post #11

I use it for a trading bot on Fifa 2020 Fut mode

I imagine you don't want to share too much info about this, but I'd be interested in any more detail you are willing to share.

Re: Ask HN: What are you using headless Chrome for?

#15

Post by enlightens » Thu, Sep 24, 2020, 10:05 PM UTC

Generating diagrams from Markdown files using the Mermaid JS library. They have a CLI but it's NPM based so you basically have to pick your poison as to what you'll be configuring. In my case Chrome made more sense.

Re: Ask HN: What are you using headless Chrome for?

#16

Post by r0s » Fri, Sep 25, 2020, 4:30 PM UTC

Automated testing.

Re: Ask HN: What are you using headless Chrome for?

#17

Post by kypro » Fri, Sep 25, 2020, 8:15 PM UTC

eberkund wrote: post #2

I used it for creating PDFs at my last job.

Could someone elaborate? How does this work?

Do you create a HTML document then print the output as a PDF with headless Chrome?

Re: Ask HN: What are you using headless Chrome for?

#18

Post by peteypao » Fri, Sep 25, 2020, 8:26 PM UTC

jamieweb wrote: post #5

I use it to test my Content-Security-Policy by crawling my whole site and capturing any violation reports that are sent.

What's wrong with just plain old curl?

Re: Ask HN: What are you using headless Chrome for?

#19

Post by kimown » Sun, Sep 27, 2020, 1:47 AM UTC

Generate videos.

Re: Ask HN: What are you using headless Chrome for?

#20

Post by jamieweb » Sun, Sep 27, 2020, 1:04 PM UTC

peteypao wrote: post #18

jamieweb wrote: post #5

I use it to test my Content-Security-Policy by crawling my whole site and capturing any violation reports that are sent.

What's wrong with just plain old curl?

cURL doesn't load subresources or honour Content-Security-Policy headers as this is beyond the scope of what it's supposed to do.