Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HTTrack Website Copier

github.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: HTTrack Website Copier

#11

Post by alganet »

freedomben wrote: post #8
alganet wrote: post #7

Earlier quoted context omitted.

Nope. It is for the classic web (the only websites worth saving anyway).

Even for classic web, if it's behind cloudflare, then HTTrack no longer works. It's a sad point to be at. Fortunately, the single file extension still works really well for single pages, even when they are built dynamically by JavaScript on the client side. There isn't a solution for cloning an entire site though, at least that I know of

If it is cloudflare human verification, then httrack will have an issue. But in the end it's just a cookie, you can use a browser with JS to grab the cookie, then feed it to httrack headers.

If cloudflare ddos protection is an issue, you can throttle httrack requests.

Re: HTTrack Website Copier

#12

Post by smashed »

Felk wrote: post #9

Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…

I've tried both in order to archive EOL websites and I've had better luck with wget, it seems to recognize more links/resources and do a better job so it was probably not a bad choice.

Re: HTTrack Website Copier

#13

Post by dspillett »

dark-star wrote: post #4

oh wow that brings back memories. I have used httrack in the late 90s and early 2000's to mirror interesting websites from the early internet, over a modem connection (and early DSL) Good to know they're still around, however, now that the web is much more dynamic I guess it's not as useful anymore as it was back then

> now that the web is much more dynamic I guess it's not as useful anymore as it was back then

Also less useful because the web is so easy to access, I remember using it back then to draw things down over the university link for reference in my room (1st year, no network access at all in rooms) or house (or per-minute costed modem access).

Sites can vanish easily of course still these days, so having a local copy could be a bonus, but they just as likely go out of date or get replaced, and if not are usually archived elsewhere already.

Re: HTTrack Website Copier

#14

Post by _lvbh »

alganet wrote: post #11
freedomben wrote: post #8

Earlier quoted context omitted.

Even for classic web, if it's behind cloudflare, then HTTrack no longer works. It's a sad point to be at. Fortunately, the single file extension still works really well for single pages, even when they are built dynamically by JavaScript on the client side. There isn't a solution for cloning an entire site though, at least that I know of

If it is cloudflare human verification, then httrack will have an issue. But in the end it's just a cookie, you can use a browser with JS to grab the cookie, then feed it to httrack headers. If cloudflare ddos protection is an issue, you can throttle httrack requests.

> you can use a browser with JS to grab the cookie, then feed it to httrack headers

They also check your user agent, IP and JA3 fingerprint (and ensures it matches with the one that got the cookie) so it's not as simple as copying some cookies. This might just be for paying customers though since it doesn't do such heavy checks for some sites

Re: HTTrack Website Copier

#15

Post by rkhassen9 »

corinroyal wrote: post #6

One time I was trying to create an offline backup of a botanical medicine site for my studies. Somehow I turned off depth of link checking and made it follow offsite links. I forgot about it. A few days later the machine crashed due to a full disk from trying to cram as much of the WWW as it could on there.

That is awesome.

Re: HTTrack Website Copier

#18

Post by knowaveragejoe »

freedomben wrote: post #8
alganet wrote: post #7

Earlier quoted context omitted.

Nope. It is for the classic web (the only websites worth saving anyway).

Even for classic web, if it's behind cloudflare, then HTTrack no longer works. It's a sad point to be at. Fortunately, the single file extension still works really well for single pages, even when they are built dynamically by JavaScript on the client side. There isn't a solution for cloning an entire site though, at least that I know of

I'm aware of this tool, but I'm sure there are caveats in terms of "totally" cloning a website:

https://github.com/ArchiveTeam/grab-site

Re: HTTrack Website Copier

#19

Post by jregmail »

I recommend to try also https://crawler.siteone.io/ for web copying/cloning.

Real copy of the netlify.com website for demonstration: https://crawler.siteone.io/examples-exports/netlify.com/

Sample analysis of the netlify.com website, which this tool can also provide: https://crawler.siteone.io/html/2024-08-23/forever/x2-vuvb0o...

Re: HTTrack Website Copier

#20

Post by begrid »

Felk wrote: post #9

Funny seeing this here now, as I _just_ finished archiving an old MyBB PHP forum. Though I used `wget` and it took 2 weeks and 260GB of uncompressed disk space (12GB compressed with zstd), and the process was not interruptible and I had to start over each time my hard drive got full. Maybe I should have given HTTrack a shot to see how it compares. If anyone wanna know the specifics on how I used wget, I wrote it down…

wget2 has an option por paralel downloading. https://github.com/rockdaboot/wget2
Post reply on HN