221–230 of 620 posts
Say I have a friend at all the continents (including Antartica for hypthetical fun-ness), they are all willing to allocate some square meters to plunk down some servers for whatever is needed.
How would I go about it and build this small(ish) infrastructure myself?
Earlier quoted context omitted.
That seems like an odd cost increase. How do you figure it would lead to a tripling of op costs?
Lets say you have 100 TB of data, plus two backups, you are now paying for 300 TB of data.
Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
Yeah there might be some unpleasant internal conversations following that event
I thought Fortune 500 companies (at least the enterprise companies I dealt with) that had checklists for their SAAS vendors that had to check off things like Disaster Recovery Readiness, etc. At the very least this company learned a hard lesson about Disaster Recovery best practices. Hopefully all the up and coming companies reading this story learns as well. Also please remember that a backup that isn't tested IS NO…
Often the supplier has to fill out the checklist themselves.
> do you have backups?
Manager: [X]
> are they offsite?
Dev: cloud provider docs say so
Manager: [X]
Customer: Great, you won the bid.
Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…
> DO seems to have gone with the "hire cheap overseas support that almost but doesn't quite understand English" strategy, whereas the tier 1 guys at Linode have on occasion demonstrated more Linux systems administration expertise than I've got. That's simply not true. There's support engineers hired around the world, and depending on when your ticket is posted, someone awake at that time will answer. DO is super remo…
Going to guess from this tweet https://twitter.com/AntoineGrondin/status/113096281882239385... that you're currently staffed at DO. That's fine; I know two people who do great work on DO's security team. But it would be helpful if you could disclose this when you comment about your employer publicly so that readers don't have to dig up your keybase and then your twitter account to understand it.
Earlier quoted context omitted.
> In any case, it's also probably too much infrastructure cost for a 2-man company. keeping your production and your backups in the same cloud provider is the equivalent of keeping your backup tapes right next to the computer they're backing up. you're exposing them both to strongly correlated risks. you've just changed those risks from "fire, water, theft" to "provider, incompetence, security breach"
So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3? I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
> So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3?
You're paying to try and ensure you don't need to restore from backups. Our data lives in an RDS cluster (where we pay for read replicas to try and make sure we don't need to restore from backups) and in S3 (where we pay for durable storage to try and make sure we don't need to restore from backups), but none of that is a backup!
If you're not on the AWS cloud S3 is a decent place to store your backups of course, but storing your backups on S3 when you're already on AWS is, at best, negligent, while treating the durability of S3 as a form of backups is simply absurd.
> I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
The company I work for is on the AWS cloud, so we store our backups on B2 instead. It's no more work than storing them on S3, and it means we still have our data in the event that we, for whatever reason, lose access to the data we have in S3. Who the hell doesn't have offsite backups?
Earlier quoted context omitted.
I have interviewed with DO and they tried diverting me towards a support position. They told me that on a single day a support engineer was supposed to help/advice customers on pretty much whatever the customer was having issue with and also handle something between 80-120 tickets per day. It's nice to see that DO is willing to help on pretty much anything they (read: their team) has knowledge about, but with 80-120…
7 tireless hours of work (with lunch break) 15 minutes to Listen, understand and resolve an issue, assuming perfect knowledge, a lot of luck and normal human speed, that would still amount to less than 30 resolutions a day.
Earlier quoted context omitted.
> DO seems to have gone with the "hire cheap overseas support that almost but doesn't quite understand English" strategy, whereas the tier 1 guys at Linode have on occasion demonstrated more Linux systems administration expertise than I've got. That's simply not true. There's support engineers hired around the world, and depending on when your ticket is posted, someone awake at that time will answer. DO is super remo…
> That's simply not true. There's support engineers hired around the world, and depending on when your ticket is posted, someone awake at that time will answer. DO is super remote friendly and as a result, has employees (and support folks) everywhere on the planet. Not "cheap oversea support" at all. There's a lot of support folks in the US, Canada, Europe, and in India where they have a datacenter. Going to guess fr…
Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…
I've been on Linode for 8+ years now (moved there from Slicehost when Rackspace swallowed them up) and their service (not necessarily customer support) has significantly degraded. Not sure I blame them though. They've become far more popular since I started with them and are probably doing their best to grow... but I no longer recommend them as I used to. Just my experience though.