Storing Scraped Data in an SQLite Database on GitHub
1–10 of 10 posts
Re: Storing Scraped Data in an SQLite Database on GitHub
#2Presumably you can bypass the artifact retention limit by uploading them as release artifacts (which are retained forever) rather than job artifacts.
(Not that I’d advocate for this in general, since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script.)
Re: Storing Scraped Data in an SQLite Database on GitHub
#3It's fun to test the boundaries of github's services, but if you're doing something useful I'd just hire a vps, they can be had from $5 a month. You could still upload the sqlite file to github via a check-in.
Re: Storing Scraped Data in an SQLite Database on GitHub
#4Presumably you can bypass the artifact retention limit by uploading them as release artifacts (which are retained forever) rather than job artifacts. (Not that I’d advocate for this in general, since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script.)
interesting! perhaps cleaning up the older data might help abit here
> since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script
I suppose this could also be a concern with git scraping as we are bascially duplicating data through git commits (not trying to imply that one is better or worse). Having that said, I'm not sure if GitHub would be fine with any of these if more people were to do the same at a larger scale
Re: Storing Scraped Data in an SQLite Database on GitHub
#5Presumably you can bypass the artifact retention limit by uploading them as release artifacts (which are retained forever) rather than job artifacts. (Not that I’d advocate for this in general, since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script.)
interesting! perhaps cleaning up the older data might help abit here > since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script I suppose this could also be a concern with git scraping as we are bascially duplicating data through git commits (not trying to imply that one is better or worse). Having that said, I'm not sure if GitHub would be fine with any o…
What would be interesting is if you could find a way to scrape only the deltas and then somehow reconcile them into the full scrape.
Re: Storing Scraped Data in an SQLite Database on GitHub
#6Presumably you can bypass the artifact retention limit by uploading them as release artifacts (which are retained forever) rather than job artifacts. (Not that I’d advocate for this in general, since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script.)
That's exactly what I did for scraping the USCIS processing time daily: https://github.com/jzebedee/uscis
Re: Storing Scraped Data in an SQLite Database on GitHub
#7It's fun to test the boundaries of github's services, but if you're doing something useful I'd just hire a vps, they can be had from $5 a month. You could still upload the sqlite file to github via a check-in.
[deleted]
Re: Storing Scraped Data in an SQLite Database on GitHub
#8Presumably you can bypass the artifact retention limit by uploading them as release artifacts (which are retained forever) rather than job artifacts. (Not that I’d advocate for this in general, since ultimately you’re duplicating a bunch of data and will eventually catch the eye of some GitHub compliance script.)
That's exactly what I did for scraping the USCIS processing time daily: https://github.com/jzebedee/uscis
out of curiosity: is there a specific reason to use robinraju/release-downloader@v1 over actions/download-artifact@v4 here at your 'Download previous DB' step in build_db.yml?
Re: Storing Scraped Data in an SQLite Database on GitHub
#9Earlier quoted context omitted.
That's exactly what I did for scraping the USCIS processing time daily: https://github.com/jzebedee/uscis
out of curiosity: is there a specific reason to use robinraju/release-downloader@v1 over actions/download-artifact@v4 here at your 'Download previous DB' step in build_db.yml?
It's used to download the previous day's database from the release for computing the diff. actions/download-artifact would only work for artifacts created during the current run, i.e., today's database.
Re: Storing Scraped Data in an SQLite Database on GitHub
#10Earlier quoted context omitted.
That's exactly what I did for scraping the USCIS processing time daily: https://github.com/jzebedee/uscis
out of curiosity: is there a specific reason to use robinraju/release-downloader@v1 over actions/download-artifact@v4 here at your 'Download previous DB' step in build_db.yml?
Release != artifact