31–39 of 39 posts
Earlier quoted context omitted.
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
Isn't bandwidth mostly dirt cheap/free these days?
Earlier quoted context omitted.
If it is cloudflare human verification, then httrack will have an issue. But in the end it's just a cookie, you can use a browser with JS to grab the cookie, then feed it to httrack headers. If cloudflare ddos protection is an issue, you can throttle httrack requests.
> you can use a browser with JS to grab the cookie, then feed it to httrack headers They also check your user agent, IP and JA3 fingerprint (and ensures it matches with the one that got the cookie) so it's not as simple as copying some cookies. This might just be for paying customers though since it doesn't do such heavy checks for some sites
Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
> it took 2 weeks and 260GB of uncompressed disk space Is most of that data because of there being like a zillion different views and sortings of the same posts? That’s been the main difficulty for me when wanting to crawl some sites. There’s like an infinite number of permutations of URLs with different parameters because every page has a bunch of different link with auto-generated URL parameters for various things,…
In terms of possible permutations, MyBB is pretty tame thankfully. Only the forums are sortable, posts only have the regular and the aforementioned threaded mode to view them. Even the calender widget only goes from 1901-2030, otherwise wget might have crawled forever.
I originally considered excluding threaded mode using wget's `--reject-regex` and then just adding an nginx rule later to redirect any incoming such links to the normal view mode. Basically just saying "fuck it, you only get this version". That might be worth a try for your case
Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
Earlier quoted context omitted.
If it is cloudflare human verification, then httrack will have an issue. But in the end it's just a cookie, you can use a browser with JS to grab the cookie, then feed it to httrack headers. If cloudflare ddos protection is an issue, you can throttle httrack requests.
> you can use a browser with JS to grab the cookie, then feed it to httrack headers They also check your user agent, IP and JA3 fingerprint (and ensures it matches with the one that got the cookie) so it's not as simple as copying some cookies. This might just be for paying customers though since it doesn't do such heavy checks for some sites
These protections are against ddos attacks, botnets, large crawling infrastructures that can lose by having to sync header info.
If you're just a single tired dev saving a website because you care about some content, none of this is a significant barrier.
Earlier quoted context omitted.
> you can use a browser with JS to grab the cookie, then feed it to httrack headers They also check your user agent, IP and JA3 fingerprint (and ensures it matches with the one that got the cookie) so it's not as simple as copying some cookies. This might just be for paying customers though since it doesn't do such heavy checks for some sites
Dude. Cookie is a header, user agent is a header, ja3 is a header. It's the same stuff. These protections are against ddos attacks, botnets, large crawling infrastructures that can lose by having to sync header info. If you're just a single tired dev saving a website because you care about some content, none of this is a significant barrier.
Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
I've tried both in order to archive EOL websites and I've had better luck with wget, it seems to recognize more links/resources and do a better job so it was probably not a bad choice.