501–510 of 646 posts
Earlier quoted context omitted.
It's not a lot of complexity. Add object versioning for your bucket (1 click) and mirror/sync your bucket to another bucket (a few more clicks). Yes, your S3 costs will double, but usually they're peanuts compared to all the other costs, anyway. Debating it takes longer than configuring it.
As I understand it, Aeonflux was talking about redundant backups outside of S3, which are much more complex.
Earlier quoted context omitted.
Self-managed infrastructure doesn’t fail now?
We’re going to be having this same tired, pedantic, round-about conversation when Tesla’s routinely decide to take out a family of four because it mistook a plastic bag for an off-ramp. Commenters will show up like clockwork and say shit like: “What man, it’s not like cars didn’t crash before? Haha” Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.
AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
As others have said, they are not being forthright about the severity of the issue, as is standard.
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experi…
Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables?
If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses!
[0] https://www.merchantmaverick.com/learning-terminated-merchan...
Earlier quoted context omitted.
Why not both? I just got a Synology NAS and it makes cloud sync dead simple. Now the most important things are on my PC, mirrored on 2 drives in my NAS, and on AWS S3 (or any other cloud storage).
Oh yeah. My plan is to migrate everything to the NAS, then have that back up to Glacier and/or Rsync.net. By S3, do you mean Glacier?
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down.
I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type.
Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.
Earlier quoted context omitted.
I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.
Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?
And this is of course valid reason to ignore basic data preservation approaches.
Myself I am an old fart and I realize that I am too independent / cautious. But I see way too many young programmers who just read sales pitch and honestly believe that once data is on Amazon/Azure/Google it is automatically safe, their apps are automatically scalable, etc. etc.
Earlier quoted context omitted.
> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…
I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…
My building has a natural gas backup generator.
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, eve…
AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions.
Building your own datacenter is still too much like maintaining a muscle car, fiddly bits and grease under your fingernails all the time, meanwhile the world has moved on, and we now have several options in soccer mom EVs that can challenge a classic Corvette in the quarter mile, and obliterate its 0-60-0 time. There is no Hyundai for the operations people, and there should be.
I don't know the physics of shipping such a thing, but I think we really do need to be able to buy a populated and pre-wired rack and slot it into the data center. Literally slot it in. If you've ever been curious about maritime shipping, you know that they have a system for securing containers to cranes, trailers, each other, and I don't see a reason you couldn't steal that same design for mounting a server rack to the floor. Other than the pins would need to be removable (eg, a bolt that screws into a threaded hole in the floor) so you don't trip on them.
In a word, we need to make physical servers fungible. There are any number of things that we need to do to get there, but I think we can. Honestly I'm surprised we haven't heard more of this sort of talk from Dell, especially after they bought VMWare. This just seems like a huge failure of imagination. Or maybe it's simply a revolution lacking a poster child. At this rate that 'child' has already been born, and we are just waiting to see who it is.