I always wonder if this gives false positives for people just using the same WordPress template.
HTTrack Website Copier
21–30 of 39 posts
Re: HTTrack Website Copier
#22The archive saved in HTTrack Website Copier can be opened in https://replayweb.page locally or they have different save formats?
Re: HTTrack Website Copier
#23Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
Re: HTTrack Website Copier
#24This saved me a ton when back in college in rural India without Internet in 2015. I would download whole websites from a nearby library and read at home.
I've read py4e, ostep, Pgs essays using this.
I am who I am because of httrack. Thank you
Re: HTTrack Website Copier
#25Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
If you want to customize the scraping, there's scrapy python framework. You would always need to download the html though.
Re: HTTrack Website Copier
#26Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
Isn't bandwidth mostly dirt cheap/free these days?
Re: HTTrack Website Copier
#27Earlier quoted context omitted.
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
Isn't bandwidth mostly dirt cheap/free these days?
It's inexpensive, but sometimes not free. For example, Google Cloud Hosting is $0.14 / GB so 260 GB would be around $36.
Re: HTTrack Website Copier
#28Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
Is there a friendly way to do this? I'd feel bad burning through hundreds of gigabytes of bandwidth for a non-corporate site. Would a database snapshot be as useful?
MyBB PHP forums have a web interface through which one can download the database as a single .sql file.
It will most likely be a mess, depending on the addons that were installed on the forum.
Re: HTTrack Website Copier
#29Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…
> it took 2 weeks and 260GB of uncompressed disk space
Is most of that data because of there being like a zillion different views and sortings of the same posts? That’s been the main difficulty for me when wanting to crawl some sites. There’s like an infinite number of permutations of URLs with different parameters because every page has a bunch of different link with auto-generated URL parameters for various things, that results in often retrieving the same data over and over and over again throughout an attempted crawl. And sometimes URL parameters are needed and sometimes not so it’s not like you can just strip all URL parameters either.
So then you start adding things to your crawler like, starting with shortest URLs first, and then maybe you make it so whenever you pick the next URL to visit it will take one that is most different from what you’ve seen so far. And after that you start adding super specific rules for different paths of a specific site.
Re: HTTrack Website Copier
#30I have tried the windows version 2 years ago. The site I copied was our on-prem issue tracker (fogbugz) that we replaced.
HTTrack did not work because of too much javascript rendering, and I could not figure out how to make it login.
What I ended up doing was embedding a browser (WebView2) in a C# Desktop app. You can intercept all the images/css, and after the Javascript rendering was complete, write out the DOM content to a html file.
Also nice is that you can login by hand if needed, and you can generate all urls from code.