Amazon is a turn and burn employer at its heart. I've watched dozens of friends and acquaintances work there, rarely making it past 2 years, only to leave for less caustic organization thereafter. Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management, your time is limited by the up and out mentality of the organization as a whole. Amazon has created the perfect sc…
I can't really speak to amazon retail, but in my 7 odd years in aws I found most people left because * working in infrastructure means doing a lot not quite so exciting things. most people aren't actually excited by ops, and a lot of your dev work will be on ops. its also the case that ops was historically not as well rewarded, though thats been fixed * every service needs to be held to the highest tier of operationa…
> was a lot slower than than when i joined due to so much risk aversion and bureaucracy
I feel like this is true for every single large company I’ve ever worked for. Things always feel like they’re regressing from a high point. I have no idea why it it.
All of them had free coffee when I joined, and they’ve slowly removed the vending machines and just never replaced them.
Amazon is a turn and burn employer at its heart. I've watched dozens of friends and acquaintances work there, rarely making it past 2 years, only to leave for less caustic organization thereafter. Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management, your time is limited by the up and out mentality of the organization as a whole. Amazon has created the perfect sc…
It's weird how it's always the people that talk about friends that worked there but don't actually work there themselves that are usually the only ones to have negative things to tell. I've been there 5 years, haven't worked more than 40 hours a week ever, and a lot of the people that started at the same time as me are in the same situation. And of those that aren't, they left because they wanted new experiences, not…
We had a colleague moving to AWS, but he came running back within a year :P
It can’t be all bad, but it’s clearly not all good either.
It kind of makes sense that the people still there wouldn’t have bad stories to tell. What do you think is the worst part about working for Amazon?
It's weird how it's always the people that talk about friends that worked there but don't actually work there themselves that are usually the only ones to have negative things to tell. I've been there 5 years, haven't worked more than 40 hours a week ever, and a lot of the people that started at the same time as me are in the same situation. And of those that aren't, they left because they wanted new experiences, not…
We had a colleague moving to AWS, but he came running back within a year :P It can’t be all bad, but it’s clearly not all good either. It kind of makes sense that the people still there wouldn’t have bad stories to tell. What do you think is the worst part about working for Amazon?
If your prior employer kept you for half a decade before you departed for Amazon, and the employer after Amazon keeps you for years, yet your stint at Amazon lasts under 2 years, that failure reflects on Amazon.
Spending the money to recruit the employee, then pay them for a year or two, only to lose this talent is a great way to light cash on fire. That ain't cheap when your talking about hiring software developers!
Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?
The VTL was invented because, at the time (2012ish), none of the largest enterprise backup solutions had a good S3 interface and none supported Glacier. After talking with the backup software vendors (Spectrum, Tivoli, Symantec, Commvault, etc) it became clear that adding another backup target wasn't something we (AWS) could get them to prioritize, for perfectly reasonable reasons. We could (and did) apply pressure v…
You did such a good job, that when I worked on Glacier (2013-2016), our interactions with the team running VTL was already pretty minimal. It was one of the least problematical things that interacted with Glacier.
Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.
I can't really speak to amazon retail, but in my 7 odd years in aws I found most people left because * working in infrastructure means doing a lot not quite so exciting things. most people aren't actually excited by ops, and a lot of your dev work will be on ops. its also the case that ops was historically not as well rewarded, though thats been fixed * every service needs to be held to the highest tier of operationa…
> was a lot slower than than when i joined due to so much risk aversion and bureaucracy I feel like this is true for every single large company I’ve ever worked for. Things always feel like they’re regressing from a high point. I have no idea why it it. All of them had free coffee when I joined, and they’ve slowly removed the vending machines and just never replaced them.
The team working on developing integrations for Amazon Payments certainly didn't have diddly squat in their break room when I visited a few years back. Not revenue generating = Get treated terribly at Amazon was a sight to behold in action.
I think I'll point to this as one of the examples of how AWS has been scrappy and successful. (Yes, I'll get like 10 flames on this comment.) They made this nice simple API for S3 that has become a de facto standard. They added lots of features and security controls, and I can easily imagine an architect saying "everyone should just adopt S3, it's so easy!" But huge, paying customers were used to tape libraries so th…
> though it's hard to beat the durability and cost of LTO tape for long-term archival I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies. Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later. Likewise, AM radio may not…
It's very unlikely that you'll be able to read modern tape from closet in 50 years. Tape requires very precise temperature and humidity for storage. Otherwise all bets are off.
Only one datapoint; but I’ve seen an FPS/Celerity minicomputer boot off of a Qic-20 tape that’d been stored in a non climate controlled Albuquerque storage unit for 19 years…
Amazon has a service to send a semi-trailer to the loading dock of your data warehouse, so you can forklift in pallets of physical media. Then hauls it all away for ingestion into its cloud. Or, if it will fit in your station wagon, you can haul your data down the highway to one of Amazon's clouds. Or put it in a box and ship it Fedex.
That's not quite how it works. Amazon will send you various storage appliances (including a semi hauling a 45ft shipping container full of hard drives), which you connect to your network, upload your data, and send it back to them. They don't accept pallets of arbitrary media devices. https://aws.amazon.com/snow/
Thanks.
I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.
Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?
From the perspective of the AWS solution, this is a way of giving on-premises hardware an option versus Iron Mountain or similar for archiving to offsite for DR/BC purposes. Since AWS' storage options are typically pretty insane SLAs, this is acceptable (and certainly not much worse than having old LTO tapes in a storage locker).
VTLs have been around for a long, long time. Initially, they were a way of speeding up tape by writing to a much faster media, and allowing for deduplication on the media to reduce storage costs. This was, of course, in the days of on-prem arrays and storage. Over time, LTO (the dominant tape format, I think we're on LTO-9?) became denser and faster, and the speeds at which you could write to a single drive would outpace the line speeds. Given that, methods were introduced to interleave multiple writers to a single drive/cart, and to leverage the drives speed. One thing you don't want on a tape drive is "shoeshining", this is the effect when the drive has no data to write, but the momentum of the reels in the drive mechanism carry the media forward, past the last written segment, past the write head. The drive must then backup and re index to find where it left off while queuing whatever data is incoming.
Well, as VTLs were built initially to handle speed of writes and dedup, once the tape drives outpaced those issues, VTL became problematic. It was basically taking a randomly writeable media and turning it into a serial storage mechanism. This really wasn't much use, and VTL started to fade. Many solutions were built, one from NetBackup was OpenStorage; this was an API that a storage vendor could write a plugin for and allow NetBackup to write to their storage as intended: multiple writer, multiple access. Allowing for treatment of a deduplicable storage media as a filesystem versus an emulated tape drive. This API approach allowed for plugins to be constructed to talk to any storage, including cloud.
AWS spoke with NetBackup at one point, intending to build a storage mechanism for a very solid solution: allow on-prem customers to export their data to AWS storage, AND to recover that data BACK from AWS storage (think of the data transfer fees you would get out of that!!). When they looked at the options, they decided to build the VTL gateway. I'm not sure of how or where the decision was made, but it seemed a bit odd at the time. As VTLs were fading and the limitations of emulating tape would lead to a caching requirement that might be bad, but with line speeds for interconnect to AWS making the caching requirements even worse. I'm not quire sure how that was done, but I will say, a VTL will always be a stable, solid thing, that tends to simply run and do what was intended; much like a traditional tape drive.
As for some of the other discussion here, about reading old tapes and storage longevity; I've seen tapes kept for decades that were readable, Iron Mountain is still in business for a reason. Note that THEY don't have a solid cloud offering, but really, if you had everyones old data sitting around on tapes, and could charge not only for data delivery but media conversion at DR time, wouldn't you?
And the issue with old tapes isn't so much the storage in that situation, it's the hardware. LTO only reads 2 generations backward (IIRC), and can only write one gen forwards. So, for your LTO-6 drive, it can read LTO-4 to 6, and can write LTO-6 to 7, again if memory serves. So, finding the Apollo code tapes on a format tape where the only possible drives available to read it live in a museum and may or may not actually work, make the fact that they recovered that source code amazing. Not to mention the software needed to be built to interface to hardware that the last person to build a "driver" for likely is sitting in a nice retirement joint in Sarasota.