Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Tape Gateway

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 139 posts

Re: AWS Tape Gateway

#51

Post by artificialLimbs »

thomaslord wrote: post #35
floren wrote: post #27

Earlier quoted context omitted.

Backblaze B2 would cost you about $10/mo to store 2TB. I've been using them for years for a few tens of gigabytes worth of documents and photos. https://www.backblaze.com/b2/cloud-storage-pricing.html edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS a…

Have you had any luck with picking and choosing what gets backed up? I have a bunch of smaller personal files I'd love to sync to B2 (photos, backups, etc.) but I also have 7TB+ of linux ISOs that don't really need to be backed up because I can easily download them again. I tried to exclude the folders I don't want to back up, but I haven't had any luck.

I use rsync with Backblaze. I put everything I want sent to BB in /backupfolder/Archive, and everything else straight in /backupfolder. I have 2 rclone remotes, one pointing to another machine at my house, and 1 pointed to BB.

Re: AWS Tape Gateway

#52

Post by lopkeny12ko »

kcb wrote: post #48
jurassic wrote: post #23

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed…

There is rigor behind the claim. Check out this talk from a few years ago: https://www.youtube.com/watch?v=DzRyrvUF-C0

Re: AWS Tape Gateway

#53

Post by orf »

logifail wrote: post #49
thedougd wrote: post #7

Earlier quoted context omitted.

It's the latter. In a large enterprise, the backup configuration can be wildly complicated, a matrix of systems, schedules, slas, etc. Just reconfigure your backup software to use this new virtual tape device and you're on your way.

> Just reconfigure your backup software to use this new virtual tape device and you're on your way Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz. "Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences. Then "we got hacked and our local backups are…

Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually all the tapes where stored at the wrong temperature or humidity and now they are fucked" -> ... ??

Re: AWS Tape Gateway

#54

Post by tablespoon »

sithadmin wrote: post #8
SCdF wrote: post #5

Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?

The gateway appliance is fairly compatible with legacy backup solutions, which makes it a great drop-in replacement for physical tape library systems. There are certainly better backup methods available these days (though it's hard to beat the durability and cost of LTO tape for long-term archival), but I've seen AWS's virtual tape used as a good stopgap while other backup/recovery solutions are still a ways out an o…

> though it's hard to beat the durability and cost of LTO tape for long-term archival

I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies.

Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later.

Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).

Re: AWS Tape Gateway

#55

Post by chaxor »

kcb wrote: post #48
jurassic wrote: post #23

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed…

One bit change in a sea of 100TB is quite a small percentage. Could work that way.

Re: AWS Tape Gateway

#56

Post by sshagent »

shrike wrote: post #20
SCdF wrote: post #5

Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?

The VTL was invented because, at the time (2012ish), none of the largest enterprise backup solutions had a good S3 interface and none supported Glacier. After talking with the backup software vendors (Spectrum, Tivoli, Symantec, Commvault, etc) it became clear that adding another backup target wasn't something we (AWS) could get them to prioritize, for perfectly reasonable reasons. We could (and did) apply pressure v…

You did such a great job they still try and steer people this route. I've multiple times mentioned "you know we don't need to emulate tape drives any more"

Re: AWS Tape Gateway

#57

Post by ghshephard »

glasss wrote: post #10

My first and only experience with tape backups was in 2014 working with the BBB of Chicago. They were using tape backups, and there I learned that almost no company that small should be using tapes. Unless you have one or two people dedicated to handling that plus whatever other backup solutions you have in place, it won't end well.

Agreed you need someone dedicated a minimum 1/2 to full headcount to run a backup tape backup system - particularly the first six months when you are getting all your routines in place (if can drop down to a minimum 8-10 hour/week time commitment once things are running smoothly - but obvious can scale up to many many headcount if you have a lot of data and a lot of systems being backed up). You purchase a Tape Library with sufficient drives / slots to support your data volume, you decide on tape software, you set up a pickup schedule with Iron Mountain or whoever will pick up (and return) your tapes, set up up your schedule of fulls / diffs rotation. There are three tricky and time consuming elements that require a lot of knowledge and dedicated time - #1 making sure you have a consistent snapshot and backup of the snapshots and RESTORE process for all your systems. Someone needs to test/verify roughly quarterly test restores (very time consuming and annoying but critical). And, the one thing that very few people do, but is also important - doing a “Black Start” - in which your primary backup site (with the robot, and software, and systems) are lost and you have to bootstrap, from scratch everything.

So, yeah - virtual tape library, while obviously having downsides, is massively useful for a small enterprise.

Re: AWS Tape Gateway

#58

Post by daxfohl »

kcb wrote: post #48
jurassic wrote: post #23

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed…

Yeah, the most depressing part of my job is when I have to calculate the SLO metrics for our directors. The first pass of stuff we control is always way within SLO. Then we have to subtract stuff where some third party messes up, or like the request doesn't even get to our service and it always puts us in the red.

Re: AWS Tape Gateway

#59

Post by guenthert »

robohoe wrote: post #33
glasss wrote: post #10

My first and only experience with tape backups was in 2014 working with the BBB of Chicago. They were using tape backups, and there I learned that almost no company that small should be using tapes. Unless you have one or two people dedicated to handling that plus whatever other backup solutions you have in place, it won't end well.

I hear you. I had to manage a Quantum LTO4 tape library with Bacula back in early 2010s at a small company. Yeah that was a fun experience (not) and I'm glad I don't have to deal with them. Good riddance. Just getting the labeling right and making sure that the tapes were writable was a pain.

But that is Bacula fault isn't it? I used it myself @home and couldn't help thinking that there must be a better way. At work Commvault was used iirc, but others were in charge of that. I evaluated Tivoli SM, but that's more than 15 years ago and I barely remember. TSM and Bacula share some concepts, but the latter appears quite hackish. Not that confidence inspiring.
Post reply on HN