Live data from Hacker News

Show HN: Baxx – Unix-friendly backup service

txt.black

191–200 of 203 posts

Re: Show HN: Baxx – Unix-friendly backup service

#191
post #141

Earlier quoted context omitted.

I evaluated Borg and Restic and found that both of them fall over once you get to (what I consider to be) production level volumes; in my case that's ~1 PB and in the range of a billion files. Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

I can't say i've been testing with PB sizes, but for my "meager" 8TB backup, Borg works well. A daily backup & prune operation takes less than 20 minutes. Restic falls over as soon as you cross 2TB sizes, and prune operations are painfully slow. Backing up 8TB took 4 weeks with Restic, and i gave up waiting for prune to finish. It was crossing the 24 hour mark, making it unsuitable for daily backups.

Yeah. I back up a few terabytes a day with no problem in Bacula.

Re: Show HN: Baxx – Unix-friendly backup service

#192
post #183

(Alternative product recommendation, please downvote/remove if you feel that isn't appropriate) For Unix/Linux backups, may I suggest Borg Backup? It encrypts and does dedupe astonishingly well. It also works over SSH incredibly fast, and restores are via a mounted FUSE filesystem so they're easy to pick and choose what you need. It prunes really well too, and is a single executable so it's easy to distribute via Ans…

For bash scripting: have you seen shellcheck? It's a very solid first-pass for any script, and the codes it emits have excellent documentation: https://www.shellcheck.net/

Yes I have, I made sure it passes shellcheck before posting it here :)

I do sort of wish that Visual Studio Code had the "auto-fixes" already built for shellcheck's error messages. I should probably create those and make them available somewhere.

Re: Show HN: Baxx – Unix-friendly backup service

#193
post #161

(Alternative product recommendation, please downvote/remove if you feel that isn't appropriate) For Unix/Linux backups, may I suggest Borg Backup? It encrypts and does dedupe astonishingly well. It also works over SSH incredibly fast, and restores are via a mounted FUSE filesystem so they're easy to pick and choose what you need. It prunes really well too, and is a single executable so it's easy to distribute via Ans…

You should make a habit of setting set -euo Pipefail if "${DEBUG:-false}”; then set -x; fi The -o Pipefail is irrelevant in this case, as you're not using any.. but better it's there in case you ever extend the script. As others have pointed out: readability comes right after correctness in shell script, do keeping it simple as you have done is always a good idea.

I hope you see this reply, I am sorry for it being so late:

Do you have a link to a good explanation of why you'd set those? The "set -e" seems to indicate that the script would fail immediately if the borg backup job fails, which would prevent the Zabbix "send" command from alerting the monitoring system.

Thank you for your feedback, it's given me more to learn and think about!

Re: Show HN: Baxx – Unix-friendly backup service

#194
post #161

Earlier quoted context omitted.

You should make a habit of setting set -euo Pipefail if "${DEBUG:-false}”; then set -x; fi The -o Pipefail is irrelevant in this case, as you're not using any.. but better it's there in case you ever extend the script. As others have pointed out: readability comes right after correctness in shell script, do keeping it simple as you have done is always a good idea.

I hope you see this reply, I am sorry for it being so late: Do you have a link to a good explanation of why you'd set those? The "set -e" seems to indicate that the script would fail immediately if the borg backup job fails, which would prevent the Zabbix "send" command from alerting the monitoring system. Thank you for your feedback, it's given me more to learn and think about!

There are quite a few examples around, just google the parameters 'set euo pipefail' (first hit [0])

you're right that just putting the set options on top of your file would be a bad idea. you need to write your script to actually account for these errors, making possible exitcodes entirely transparent

    1 #!/bin/bash
    2
    3 set -euo pipefail
    4 if ${DEBUG:-false}; then set -x; fi
    5
    6 function random_exitcode(){
    7         return $(( RANDOM % 5))
    8 }
    9
   10 function handle_error(){
   11   case "$?" in
   12     1)  echo "handling exitcode 1!";;
   13     2)  echo "handling exitcode 2!!";;
   14     3)  echo "handling exitcode 3!!!";;
   15     *)  echo "encountered unexpected exitcode: $?"; exit 2;;
   16   esac
   17 }
   18
   19 random_exitcode || handle_error
or, if you don't like functions:

  random_exitcode || LAST_EXITCODE=$?
  case "${LAST_EXITCODE:=0}" in ...

[0] https://coderwall.com/p/fkfaqq/safer-bash-scripts-with-set-e...

Re: Show HN: Baxx – Unix-friendly backup service

#195
post #194

Earlier quoted context omitted.

I hope you see this reply, I am sorry for it being so late: Do you have a link to a good explanation of why you'd set those? The "set -e" seems to indicate that the script would fail immediately if the borg backup job fails, which would prevent the Zabbix "send" command from alerting the monitoring system. Thank you for your feedback, it's given me more to learn and think about!

There are quite a few examples around, just google the parameters 'set euo pipefail' (first hit [0]) you're right that just putting the set options on top of your file would be a bad idea. you need to write your script to actually account for these errors, making possible exitcodes entirely transparent 1 #!/bin/bash 2 3 set -euo pipefail 4 if ${DEBUG:-false}; then set -x; fi 5 6 function random_exitcode(){ 7 return $…

Many thanks, I very much appreciate the effort you put into your reply, and your followup.

Re: Show HN: Baxx – Unix-friendly backup service

#196
post #132

Two reasons why I do not want to use the service in its current state (meant as constructive criticism): 1. If I would send my backups to some SASS there is no way I would do that without encryption. 2. I like to backup my filesystem and not just the files in it (to make sure I've got everything and make restoring easy). Currently, I just dd my block devices, but I am sure that could be optimized to not upload a comp…

> Currently, I just dd my block devices That's actually not a good idea at all. It's very fragile, the slightest problem and you may lose the entire backup. Silent corruption may get backed up for months and you won't notice until it's too late. Doing a restore means having space for the full file, just to restore a single small file. If you don't know when your file changed you may have to do that multiple times ins…

Why is dd fragile? I mean, yes doing so while the filesystem is mounted is like Russian Roulette, but if the filesystem is unmounted? I see that it isn't very efficient, but I never had any problems with reliability.

Silent corruption might actually be a problem, but that is something you won't solve by backing up files and directories instead of block devices. After all, a block devices backup just contains more information than the files and directories backup. The only advantage of filesystem backups is that you can easier validate if a file should have changed, but even if you detect that it changed even if it shouldn't have, you still need some kind of checksum or so to find out which version is correct.

On the other hand, if you back up the files and directories you have to care about the filesystem type and if there are special types like links, devices nodes and the like. On that side, I had enough unpleasant experiences that I am trying to avoid that trouble.

To restore single files I can simply mount the image on a loop device, so no problem there.

Re: Show HN: Baxx – Unix-friendly backup service

#197
post #171
post #168

Earlier quoted context omitted.

In that case, what do you use?

We're using Bacula, because in our case it's the only thing that works with our volume, and also works with tape (which is the only cost effective way we've found to archive multi petabyte datasets). But I'm not very happy about it, because it's insanely overcomplicated for no really good reason, and because of it being file-based.

Backblaze has done some cost comparisons between LTO and cloud storage:

https://www.backblaze.com/blog/lto-versus-cloud-storage/

If you're interested, I'm doing experiments with another site with 500T to backup, where I added sampling and sharding to HashBackup (I'm the author).

Sampling allows you to do faster simulated backups to determine the best backup parameters to use. In his case, we determine that a very large block size - 64M - was the best way to backup his data.

Sharding automatically partitions the filesystem so multiple backup can run simultaneously to get backup speed in the 250-400 MB/s range.

It's more at the proof of concept stage, but having another large site to work with would be fantastic! A couple of the larger sites using HashBackup are EURAC (European Research Center) and HMDC (Harvard MIT Data Center)

Re: Show HN: Baxx – Unix-friendly backup service

#198
post #197
post #171

Earlier quoted context omitted.

We're using Bacula, because in our case it's the only thing that works with our volume, and also works with tape (which is the only cost effective way we've found to archive multi petabyte datasets). But I'm not very happy about it, because it's insanely overcomplicated for no really good reason, and because of it being file-based.

Backblaze has done some cost comparisons between LTO and cloud storage: https://www.backblaze.com/blog/lto-versus-cloud-storage/ If you're interested, I'm doing experiments with another site with 500T to backup, where I added sampling and sharding to HashBackup (I'm the author). Sampling allows you to do faster simulated backups to determine the best backup parameters to use. In his case, we determine that a very lar…

Yeah, I read that Backblaze thing, and used their spreadsheet. Tape came out to be nearly 9x cheaper for us, given our large data set but extremely slow growth rate. Basically, we have a lot of data we need to protect from disaster, but it changes/gets added to very slowly, so capex (buy some tapes, put the data on tape, put the tapes in an offsite box and forget about them) is vastly cheaper than opex (pay by the month for hundreds of terabytes).

Re: Show HN: Baxx – Unix-friendly backup service

#199
post #131

This is really neat. I really love the idea and the presentation. I would not trust my backups to your service yet , just because of the "this is a prototype" language. My immediate thought is, "this seems great, I'll have to come back and check it out once it's more of a real business". But therein lies the rub, I think: what will drive me back to check it out later? There doesn't seem to be a mailing list to sign u…

good point, I added 'make a mailing list' to the todo

i made a mailing list and slack

  slack: https://baxx.dev/join/slack
  google groups: https://baxx.dev/join/groups

Re: Show HN: Baxx – Unix-friendly backup service

#200
post #34

I get that it's probably "trendy" to say you don't have or need a website but with it being so easy to create a static page, I'm failing to understand why you wouldn't just do that, which is already the bare minimum, if you care about your project and want to sell it?

Had he done that I doubt it would have done so well on HN. It’s a differentiation point from yet another bootstrap css SaaS with three tier pricing, faded monotone brand logos and circlular portrait photos of the “team”.

tbh i just wanted to have fun building it having the ssh registration just makes me excited, i feel happy every time i try it out; using the api and playing with it is the same, like making a todo list on baxx https://github.com/jackdoe/baxx/blob/master/examples/todo.sh

to me the `yet another bootstrap css SaaS with three tier pricing, faded monotone brand logos and circlular portrait photos of the “team”` is just depressing

Post reply on HN