Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The wrong way to switch operating systems on your server

figbert.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: The wrong way to switch operating systems on your server

#61

Post by Dylan16807 »

wildmanx wrote: post #17

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blind…

> After _that_ experience, he blindly trusts the result of _the same tool_, blindly wipes everything? Wtf.

You imply that there's something wrong with the tool because he aborted early the first time? Why?

I would check the backup too but why emphasize "the same tool" like that?

It has a -v flag.

Re: The wrong way to switch operating systems on your server

#62

Post by [deleted] »

wolrah wrote: post #50
rekabis wrote: post #49

One big WTF in my mind is the default lack of feedback with tarsnap. The -v should be implicit and automatic , and should need a flag to turn OFF, not on. I mean, that’s just UI/UX 101. _Always_ provide feedback on progress, unless explicitly and intentionally silenced.

Tools that are intended to be used in scripts often have fairly quiet default modes. tar, cp, and many other *nix tools in that general realm tend to only speak up when something goes wrong, at least by default. That said, I'm on your side and I think quiet mode should be the option with a reasonable human-friendly verbosity being default. Or even better, some form of detection of if it's being run interactively vers…

[deleted]

Re: The wrong way to switch operating systems on your server

#63

Post by Dylan16807 »

geofft wrote: post #39

> My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous. > What if it failed silently? How can I check? What should I do? On Linux, find the process ID and run e.g. ls -l /proc/12345/fd which will show you all the files currently open by the process. For something like a backup of a whole directory, or something generating a lot of output, run it ag…

Also there's the simple method of using iotop or a network equivalent.

Re: The wrong way to switch operating systems on your server

#64

Post by neurostimulant »

klodolph wrote: post #4

These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons here. You want to switch operating systems on your server? 1. Set up monitoring. If you don't have monitoring already, hack something together with simple scripts. 2. Start up a new server. 3. Migrate services and data from your existing server to your new ser…

I would also setup a reverse proxy on the old server so it would forward all traffics to the new server, and finally kill the old server once the reverse proxy no longer see any legitimate requests.

Re: The wrong way to switch operating systems on your server

#66

Post by neurostimulant »

Shank wrote: post #21

Far and above, the best strategy is to spin up the new server, scp/rsync the data to the new server directly from the old server, and then boot services, and only decommission after you’ve moved all DNS over and confirmed the new site is working. Using Tarsnap for this is not only time consuming but needless unless you already have it setup and working.

My strategy is to setup the new server, and once everything is ready, stop the service on the old server and replace it with a reverse proxy to the new server, and THEN finally update the dns record. When the old server no longer see any meaningful traffic, I'll decommission the old server. This way I don't have to deal with both server out of sync.

Re: The wrong way to switch operating systems on your server

#67

Post by stavros »

I had exactly the same problems, and I decided to change not my backup strategy, but my deployment strategy. I wrote a small tool to deploy everything in a single directory, using Docker Compose:

https://gitlab.com/stavros/harbormaster

It allows you to separate important state (data) from non-important state (caches). This way, all you need to do is back up the data directory, and then you can restore the Harbormaster config file (along with the data) on the new server and you're done.

Re: The wrong way to switch operating systems on your server

#68

Post by gjm11 »

bigiain wrote: post #54
gperciva wrote: post #37

Wow, that's much more technically advanced than I was as a teenager! Way to go! To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as killall -SIGUSR1 tarsnap https://www.tarsnap.com/tips.html#check-current (Note that Tarsnap is not responsible for naming the unix `kill…

"employed by"

Note that that's gperciva not cperciva. I believe the person's name is Graham Percival, and I am fairly sure Colin Percival is still called Colin Percival. They may well be related, but I'm pretty sure that's a bona fide "employed by" :-).

Re: The wrong way to switch operating systems on your server

#69

Post by spsesk117 »

dspillett wrote: post #33
spsesk117 wrote: post #7

Earlier quoted context omitted.

+1. `strace` can be very helpful here, to see if a process is stuck waiting on something or whether it's just zooming along with no output.

`progress` ( https://github.com/Xfennec/progress ) and similar can be very helpful too depending on the backup utilities being used (in my case often involving rsync) even if the processes normally have everything set to quiet so no progress information is automatically forthcoming.

Wow this looks great, didn't know about this tool. Thanks for the recommendation!

Re: The wrong way to switch operating systems on your server

#70

Post by FpUser »

Every server of mine has "history" in a form of script that can rebuild it from the scratch on brand new clean machine. Every time anything is added / modified the script is ran on clean machine to test if all still works as supposed. My databases are all automatically backed to a different backup servers and are restored when needed by the same script.

So for me setting up new server involves following steps:

1) Modify config portion of the script (basically change default values if needed) and run said script on a new server 2) Test 3) Put old server into "read only mode" 4) Run script again with the option to restore the database only. 5) Switch DNS

There are/could be slight variations of course. Practically, I did recent migration of the new server for new product that was running on my location in Toronto to Hetzner. It took a "whopping" 15 minutes of which 13 been taken by restoring database from backup.

Post reply on HN