Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrapism

scrapism.lav.io

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Scrapism

#11

Post by sackerhews »

simonw wrote: post #4

A trick I think would be useful to include here is running scrapers in GitHub Actions that write their results back to the repository. This is free(!) to host, and the commit log gives an enormous amount of detail about how the scraped resource changed over time. I wrote more about this trick here: https://simonwillison.net/2020/Oct/9/git-scraping/ Here are 267 repos that are using it: https://github.com/topics/git-s…

Honorable mention even if he doesn't use Actions: https://github.com/elsamuko/Shirt-without-Stripes

Re: Scrapism

#13

Post by saaaam »

Hi! This is a guide that I started during the pandemic but never quite finished. I’m in the process of re-writing/re-recording some parts of it to bring it back up to date, and adding in the bits that are still missing.

Re: Scrapism

#14

Post by Labo333 »

simonw wrote: post #4

A trick I think would be useful to include here is running scrapers in GitHub Actions that write their results back to the repository. This is free(!) to host, and the commit log gives an enormous amount of detail about how the scraped resource changed over time. I wrote more about this trick here: https://simonwillison.net/2020/Oct/9/git-scraping/ Here are 267 repos that are using it: https://github.com/topics/git-s…

Interesting! Any idea of how likely Github ips are to be blocked?

Re: Scrapism

#15

Post by electroly »

simonw wrote: post #4

A trick I think would be useful to include here is running scrapers in GitHub Actions that write their results back to the repository. This is free(!) to host, and the commit log gives an enormous amount of detail about how the scraped resource changed over time. I wrote more about this trick here: https://simonwillison.net/2020/Oct/9/git-scraping/ Here are 267 repos that are using it: https://github.com/topics/git-s…

I feel like this is bad manners. The runners are a shared resource and you risk getting their IPs blacklisted by the sites you're scraping. I think a strict reading of the GitHub Actions TOS may prohibit this sort of usage, too.

> ... for example, don't use Actions as a content delivery network or as part of a serverless application ...

> Actions should not be used for: ... any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used.

> You may only access and use GitHub Actions to develop and test your application(s).

https://docs.github.com/en/site-policy/github-terms/github-t...

Re: Scrapism

#16

Post by saaaam »

1vuio0pswjnm7 wrote: post #9

This is from 2020. Besides a small change to the "Introduction to the Command Line" section, it has not been updated. Back in 2015, the author reported using CasperJS to scrape public LinkedIn profiles. The author reported this was a PITA. Here the author recommends using WebDriver implementations, e.g., chromedriver or geckodriver, in addition to scripting language frameworks such as Puppeteer and Selenium. Is scrap…

Hi - I'd be interested to hear more details about what approaches you suggest!

Re: Scrapism

#17

Post by saaaam »

simonw wrote: post #4

A trick I think would be useful to include here is running scrapers in GitHub Actions that write their results back to the repository. This is free(!) to host, and the commit log gives an enormous amount of detail about how the scraped resource changed over time. I wrote more about this trick here: https://simonwillison.net/2020/Oct/9/git-scraping/ Here are 267 repos that are using it: https://github.com/topics/git-s…

Hi Simon! I'll definitely consider adding that in. Also, I love Datasette!

Re: Scrapism

#18

Post by saaaam »

Labo333 wrote: post #12

Good guide! The "Scraping XHR" [1] explains how to inspect network requests and reproduce them with Python. I actually built har2requests [2] to automate that process! [1]: https://scrapism.lav.io/scraping-xhr/ [2]: https://github.com/louisabraham/har2requests

Thanks for sharing this - I'll check it out!

Re: Scrapism

#19

Post by CWuestefeld »

I'm bothered that this doesn't mention any of the ethics involved, such as checking the robots.txt file and so forth.

More than half of my traffic is from bots, so I'm paying something like half my operational expenses to support them. And we've had to do a lot of work to mitigate what would otherwise be DoS attacks from badly written (or badly intended!) bots. I think that at least a tip of the hat to avoiding damage would be appropriate in a piece like this.

Re: Scrapism

#20

Post by is_true »

1vuio0pswjnm7 wrote: post #9

This is from 2020. Besides a small change to the "Introduction to the Command Line" section, it has not been updated. Back in 2015, the author reported using CasperJS to scrape public LinkedIn profiles. The author reported this was a PITA. Here the author recommends using WebDriver implementations, e.g., chromedriver or geckodriver, in addition to scripting language frameworks such as Puppeteer and Selenium. Is scrap…

Sorry if I missed something, but, Which programs do you use?
Post reply on HN