Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The wrong way to switch operating systems on your server

figbert.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: The wrong way to switch operating systems on your server

#31

Post by isatty »

Why tarsnap when wasabi or backblaze would be significantly cheaper? You can just encrypt by yourselves anyway.

Also I run my own personal infra and here’s what I do:

* treat servers as cattle, not pets. This is really important. Have mandatory reboots, never be afraid of reboots.

* preferably do things with an automation method, I use ansible for n=5 but pick whatever you like

* have SOME monitoring. It’s not too hard to throw up prom+grafana so get on it early.

* VPN instead of securing internal services. Attack surface is way too high if you’ve too many services. Just throw them all behind a vpn and expose selectively. I use WireGuard.

* personally: don’t self host critical infrastructure. I can’t afford downtime on email etc so I rather just pay someone to host that. Personal infra is for fun, not a second job (and I’m an SRE).

Re: The wrong way to switch operating systems on your server

#32

Post by ajnin »

My current backup strategy is to backup the whole filesystem. I run services in actual VMs, not docker containers, with disks mounted from LVM volumes which allows me to take snapshots and back them up live without needing to shut them down. I'm using bup to do the actual backups to a server I'm keeping in my home. I wrote a few custom scripts to backup and restore servers, and keep a history of the last x days, y weeks and z months. That way it gives me more time to figure out if something's wrong, as it's hobby stuff that I'm not checking every day.

My advice for OP would be to 1/ ditch tarsnap. A backups tool that runs for hours without any feedback ? A restore tool that fails if the files are too big ? Everything extremely slow ? Just forget it. 2/ keep more than 3 days of backups. It's too short if you make a mistake, it took 3 days to recover from this one already. 3/ backup everything. Don't try to pick and choose files, you're likely to forget something, and if not now then some time later when you create a new file but forget to add it to the list of things to back up.

Re: The wrong way to switch operating systems on your server

#33

Post by dspillett »

spsesk117 wrote: post #7
justin_oaks wrote: post #6

Don't consider it a backup until you've successfully restored the data from it. The first thing I do after setting up a new data backup is test a restore of the data. Only after that will I feel confident that the backup procedure works right. In the article author's case, an attempt to restore would have caught the problem of the missing .env files and the large movie files. As for the Ctrl-C on both the backup and…

+1. `strace` can be very helpful here, to see if a process is stuck waiting on something or whether it's just zooming along with no output.

`progress` (https://github.com/Xfennec/progress) and similar can be very helpful too depending on the backup utilities being used (in my case often involving rsync) even if the processes normally have everything set to quiet so no progress information is automatically forthcoming.

Re: The wrong way to switch operating systems on your server

#34

Post by beachy »

wildmanx wrote: post #29
wildmanx wrote: post #17

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blind…

Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.

Brutal honesty can be a wonderful trait all of its own in a team however. We can assume the author is not a BS artist from his very candid self-evaluation. This is a good thing he should hang onto.

"My haphazard strategy resulted in three days of stress and frustration as I clambered to restore a self-hosting empire that I myself had reduced to ash."

Re: The wrong way to switch operating systems on your server

#35

Post by klodolph »

lisper wrote: post #22
livueta wrote: post #9

Earlier quoted context omitted.

This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together. Since the old machine was still sitting there wit…

Anything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.) The way I usually do this is that I h…

In the corporate world, this is also a great way to hold vendors accountable.

Story: Company Red hires company Blue to take over the company blog as part of a marketing initiative. Their point of contact was the marketing team at company Red, and I'm sure they thought, "We've nabbed a big customer, this is going to be great."

However, the switch from the internal blog to the external blog goes through the operations team at company Red. The operations team tells company Blue, "Here is the plan for rolling back to the internal blog at the touch of a button."

Company Blue was suddenly much more responsive to questions from company Red.

Re: The wrong way to switch operating systems on your server

#36

Post by bluehatbrit »

wildmanx wrote: post #29
wildmanx wrote: post #17

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blind…

Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.

One of the articles they link off to about choosing a different provider explains that the author is a student. I think it's reasonable to assume they don't have much experience with this type of work. Hopefully they'll learn a lot from this process and maybe from the comments here as well. Maybe lets not write someone off before we understand more about their experience and background, especially after they shared an honest account of something they screwed up and learned from.

Re: The wrong way to switch operating systems on your server

#37

Post by gperciva »

Wow, that's much more technically advanced than I was as a teenager! Way to go!

To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as

killall -SIGUSR1 tarsnap

https://www.tarsnap.com/tips.html#check-current

(Note that Tarsnap is not responsible for naming the unix `kill` or `killall` commands.)

In the unreleased git version of tarsnap, there's a `--progress-bytes SIZE` command, which prints a progress message after every SIZE bytes are processed.

As a general note: the tarsnap-users mailing list is a great place to ask for tips. As you mentioned in your lessons learned, some of the options could have helped a lot (such as `--recover`) https://www.tarsnap.com/lists.html

(Disclaimer: I'm employed by Tarsnap Backup Inc.)

Re: The wrong way to switch operating systems on your server

#38

Post by gperciva »

wildmanx wrote: post #29
wildmanx wrote: post #17

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blind…

Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.

Do you normally hire high school students at your company?

Re: The wrong way to switch operating systems on your server

#39

Post by geofft »

> My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous.

> What if it failed silently? How can I check? What should I do?

On Linux, find the process ID and run e.g.

    ls -l /proc/12345/fd
which will show you all the files currently open by the process. For something like a backup of a whole directory, or something generating a lot of output, run it again a few seconds later. If it's opened different files, then you know it's making progress and it's not stuck.

If it's something that operates on a single file, find the number corresponding to that file in the list (the file descriptor), and run e.g.

    cat /proc/12345/fd/3
which will output a "pos" field showing the position in the file, in bytes. Compare it with the actual size of the file, and also run it again a few seconds later to see how fast it's making progress.

(You can also use strace, but that slows down your program and potentially changes how it behaves in extremely unusual cases, so it isn't the first thing I'd reach for unless I really think the program is misbehaving and I want to see what it's doing in more detail. And there are tools like iostat too, but they're systemwide.)

Re: The wrong way to switch operating systems on your server

#40

Post by nick__m »

wildmanx wrote: post #29
wildmanx wrote: post #17

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blind…

Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.

Good idea to let him graduate from highschool before hiring him !
Post reply on HN