Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Baxx – Unix-friendly backup service

txt.black

Post reply on HN
Print view View original

171–180 of 203 posts

Re: Show HN: Baxx – Unix-friendly backup service

#171

Post by dmd »

zoratu wrote: post #168
dmd wrote: post #141

Earlier quoted context omitted.

I evaluated Borg and Restic and found that both of them fall over once you get to (what I consider to be) production level volumes; in my case that's ~1 PB and in the range of a billion files. Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

In that case, what do you use?

We're using Bacula, because in our case it's the only thing that works with our volume, and also works with tape (which is the only cost effective way we've found to archive multi petabyte datasets).

But I'm not very happy about it, because it's insanely overcomplicated for no really good reason, and because of it being file-based.

Re: Show HN: Baxx – Unix-friendly backup service

#172

Post by wolfhumble »

dmd wrote: post #141
bloopernova wrote: post #28

(Alternative product recommendation, please downvote/remove if you feel that isn't appropriate) For Unix/Linux backups, may I suggest Borg Backup? It encrypts and does dedupe astonishingly well. It also works over SSH incredibly fast, and restores are via a mounted FUSE filesystem so they're easy to pick and choose what you need. It prunes really well too, and is a single executable so it's easy to distribute via Ans…

I evaluated Borg and Restic and found that both of them fall over once you get to (what I consider to be) production level volumes; in my case that's ~1 PB and in the range of a billion files. Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. image files with different versions?

I really like the concept of byte level deduplication, but have often thought the price you pay for the space reduction might not be worth it considering today's network speed as well as storage sizes and prices.

Would be interesting to hear your experience concerning this!

Re: Show HN: Baxx – Unix-friendly backup service

#173

Post by dmd »

wolfhumble wrote: post #172
dmd wrote: post #141

Earlier quoted context omitted.

I evaluated Borg and Restic and found that both of them fall over once you get to (what I consider to be) production level volumes; in my case that's ~1 PB and in the range of a billion files. Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal. Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. imag…

I'm definitely looking into what I can do as far as getting logs rotated (and thus split) better, but a lot of what I back up are legacy medical systems where making any kind of change is extremely difficult both technically and politically.

Re: Show HN: Baxx – Unix-friendly backup service

#174

Post by martinml »

GordonS wrote: post #123
m3nu wrote: post #94

Earlier quoted context omitted.

I recently launched a backup service for Borg – https://www.borgbase.com . Also big on monitoring (email, pushover, webhook), as this is a common concern for backups. It just lacks terminal-based registration. :-) You could use the GraphQL API to manage everything from the command line if you really wanted.

This looks good, and competitively priced too! What's the backend cloud storage?

Just guessing here but probably Backblaze b2? It has a similar pricing.

Re: Show HN: Baxx – Unix-friendly backup service

#175

Post by millettjon »

m3nu wrote: post #94
stavros wrote: post #67

Earlier quoted context omitted.

Borg is hands-down the best backup utility I've used. I also use Borgmatic for some added niceties around checks/notifications/etc.

I recently launched a backup service for Borg – https://www.borgbase.com . Also big on monitoring (email, pushover, webhook), as this is a common concern for backups. It just lacks terminal-based registration. :-) You could use the GraphQL API to manage everything from the command line if you really wanted.

I've been using borgbase for a couple of weeks and am very happy with it.

Re: Show HN: Baxx – Unix-friendly backup service

#176

Post by wolfhumble »

dmd wrote: post #173
wolfhumble wrote: post #172

Earlier quoted context omitted.

> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal. Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. imag…

I'm definitely looking into what I can do as far as getting logs rotated (and thus split) better, but a lot of what I back up are legacy medical systems where making any kind of change is extremely difficult both technically and politically.

Okay, I see. Thanks for the feedback!

Re: Show HN: Baxx – Unix-friendly backup service

#177

Post by netok »

m3nu wrote: post #94
stavros wrote: post #67

Earlier quoted context omitted.

Borg is hands-down the best backup utility I've used. I also use Borgmatic for some added niceties around checks/notifications/etc.

I recently launched a backup service for Borg – https://www.borgbase.com . Also big on monitoring (email, pushover, webhook), as this is a common concern for backups. It just lacks terminal-based registration. :-) You could use the GraphQL API to manage everything from the command line if you really wanted.

Is there a limit to how many times I can download my backups?

Re: Show HN: Baxx – Unix-friendly backup service

#178

Post by dspillett »

dmd wrote: post #173
wolfhumble wrote: post #172

Earlier quoted context omitted.

> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal. Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. imag…

I'm definitely looking into what I can do as far as getting logs rotated (and thus split) better, but a lot of what I back up are legacy medical systems where making any kind of change is extremely difficult both technically and politically.

A slightly hacky method might be an overlay filesystem (using FUSE or similar) that mirrors the underlying filesystem for small files but represents the larger ones as smaller units (so bigfile becomes bigfile.block0000, bigfile.block0001, ...). That way only the changed block would get transferred if you one-byte change is a modify or append rather than on insert or delete.

If the backup service is using file timestamps as the only key to refresh then it would have to store the last modification date and a good hash of each block, and when the date of the file being mirrored is updated scan each block to see if there is a change there and update the stored hash and timestamps accordingly. This would need to be orchestrated to reduce the risk of temporary corruption if there are several updates and the rechecking process coincides with a backup sweep (i.e. make sure you don't present updated dates for any block until you can present them for all needed).

For restoration, you either manually concatenate the parts or have an overlay filesystem that operates in reverse: showing the smaller block files as a single large unit.

You'd have to very thoroughly test the overlays and their interaction with the backup service before risking it on important data, so it might not be something you would genuinely consider...

Re: Show HN: Baxx – Unix-friendly backup service

#179

Post by BeeOnRope »

spasche wrote: post #119
TheWiseOne wrote: post #118

Earlier quoted context omitted.

Restic also works on Windows, whereas Borg doesn't (yet), if that's a consideration.

I use it on Windows via the Linux Subsystem of Windows 10. It's not a pure native solution, but it works quite well for me.

Is "it" Borg or Restic here?

Re: Show HN: Baxx – Unix-friendly backup service

#180

Post by m3nu »

netok wrote: post #177
m3nu wrote: post #94

Earlier quoted context omitted.

I recently launched a backup service for Borg – https://www.borgbase.com . Also big on monitoring (email, pushover, webhook), as this is a common concern for backups. It just lacks terminal-based registration. :-) You could use the GraphQL API to manage everything from the command line if you really wanted.

Is there a limit to how many times I can download my backups?

No limits. Downloads are unmetered and free, as long as they look reasonable. Once you download your full backup every hour, I'll ask you for a reason probably.
Post reply on HN