[dead]
Geico repatriates work from the cloud, continues ambitious infra overhaul
51–60 of 63 posts
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#52Earlier quoted context omitted.
So when they are setting up config files for the cloud they don't cut corners? It is insane amount of work to follow safe practices to configure your cloud. I don't see that much difference compared to doing actual admin tasks.
The entire underlying layer of possible misconfigurations is absent in the cloud. Yes, the services on top of that can still be misconfigured, but you don't get access to hosts, SANs, switches, firewalls, gateways, there isn't anything for you to mess up. The shared responsibility model allows you to also pick even more robust options. But even if you were to stick to something simple, say, object storage. A bucket o…
> The entire underlying layer of possible misconfigurations is absent in the cloud.
This is true.
Let's not forget there is a whole new, quite different, layer of potential (and easy) misconfigurations that exist only in the cloud, so it balances out.
When you can accidentally expose services with a single mouse click where it used to take someone with access to the server room going in and grabbing a cable and wiring it wrong, this category of problem is a lot more common now.
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#53[deleted]
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#54Earlier quoted context omitted.
> Also, anyone in this industry long enough has been around for "Oh, we will just replace that broken piece of hardware" that ended up "WHY IS EVERYTHING ON FIRE?" because versions didn't match up, hardware was rejected I've been doing this for 25 years and I'm not sure what this means. Dell isn't going to come back to you and say "sorry but we can't fix this". With the warranty SLA worst case scenario they'll just r…
>I've been doing this for 25 years and I'm not sure what this means. Dell isn't going to come back to you and say "sorry but we can't fix this". Dell/EMC says "Hey, here is drive replacement." We do it, 2 hours later, the volume is knocked offline. Apparently, there was mismatch between backplane version, drive version and through some weird edge case, it knocked the volume offline. Yes, they fixed it, no it wasn't p…
> Dell/EMC says "Hey, here is drive replacement." We do it, 2 hours later, the volume is knocked offline. Apparently, there was mismatch between backplane version, drive version and through some weird edge case, it knocked the volume offline. Yes, they fixed it, no it wasn't pretty since a bunch of applications had to be recovered.
Anecdotal (as is my position). I can theoretically understand this happening but not only have I never seen it, such an issue would need to be escalated. That's a "this is unacceptable" high-level phone call. A call you more than likely have a chance of someone in actual authority answering because IME unless you have SERIOUS spend with big cloud you'll be lucky to make it a rung or two up sales/support.
Plus backups and redundancies that should prevent even the failure of a chassis/storage/etc from being a significant critical issue.
> their failures tend to be you twiddling your thumbs vs hair on fire on phone with the vendor trying to get it resolved
As a Founder/CTO I have the opposite take - put me and my team in a position to /do something/ vs sitting around waiting for AWS to come back whenever it decides to and while they obscure comms, don't update the fake status dashboards, etc. Meanwhile you're telling your customer "Ummm, we don't know - Amazon has a problem. When it comes back I guess it's back".
Coming from a background of telecom, healthcare, and nuclear energy I can't believe that even flies.
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#55They had an expensive, fractured, hard to maintain on-prem layout. Then they moved to the cloud. And it turned out the cloud was expensive, fractured, and hard to maintain. So they're moving to on-prem. Any bets on what's going to happen next?
The comment about "running legacy applications in the cloud was not any cheaper" stood out to me. Just moving the same legacy design into the cloud is not the optimal way to gain cost and availability improvements. If you have ever seen a data center from Azure, GCP or AWS, you will realize how difficult it will be for any company to compete in the long run. Those companies develop new generations of data center infr…
[deleted]
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#56I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment. Geico's IT will slow to a crawl in the next years due to the immense madn…
This article read more like advertisement for VP spearheading all of this.
[deleted]
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#57If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.
Our company is literally in the 3rd week of waiting for a colo to install some new RAM modules in a server. Before that we were waiting two weeks to get a new server ordered, delivered and racked. Before that we had to wait a week for them to tell us if there was available power and network ports for the new server.
That server is the main database. And yes, there is a backup server, but for reasons, the backup server isn't working as expected. So if that main server's RAM failed for good, there goes our product, for god knows how long, considering how long it's taken so far to get a second one set up.
You don't have to deal with any of that shit in the cloud. None. You just spin up a new server in 2 seconds. You don't deal with shitty hardware, or the differences between old and new hardware (besides cpu arch, and some special classes), or incompatibilities, or running out of space, or getting smart hands in your rack, or a million other things.
And that's just the hardware side. The software side of the cloud is the one million unique hosted services they offer that you can just start using immediately. No server set-up, no configuration management, it already has security baked in, it's already integrated with the other million services, etc. You just start using it, immediately, and it just works. It saves you time, complexity, maintenance, and it gives you reliability, compatibility, flexibility, and allows you to ship something earlier.
I have managed servers on-prem for years, for tiny startups and huge companies. Both two decades ago, and two years ago. Without a doubt, I would always suggest any kind of hosted, cloud-style vendor over on-prem. Only somebody needs to be on-prem, or they literally are a teenager with no money at all and all the time in the world to waste DIYing, then I would tell them to go on-prem.
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#58I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment. Geico's IT will slow to a crawl in the next years due to the immense madn…
First you charge them to put a star on their belly, and then you can charge them to take the star off their belly!
Yes!!
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#59Earlier quoted context omitted.
The entire underlying layer of possible misconfigurations is absent in the cloud. Yes, the services on top of that can still be misconfigured, but you don't get access to hosts, SANs, switches, firewalls, gateways, there isn't anything for you to mess up. The shared responsibility model allows you to also pick even more robust options. But even if you were to stick to something simple, say, object storage. A bucket o…
> The entire underlying layer of possible misconfigurations is absent in the cloud. This is true. Let's not forget there is a whole new, quite different, layer of potential (and easy) misconfigurations that exist only in the cloud, so it balances out. When you can accidentally expose services with a single mouse click where it used to take someone with access to the server room going in and grabbing a cable and wirin…
There is a middle era between a cable in a datacenter and a misclick in a cloud. Currently, on-prem is still 1 misclick away from accidental exposure (unless it's been untouched for 20+ years).
Be it with a legacy DMZ setup or a bit more segmented with a ADC/Proxy policy that is slightly too wide. You can make those exact same mistakes with a stack of PaloAlto/Cisco/F5/IIS.
Unless you're running an entire OpenStack setup with SDN layers and policies (hit: most on-prem setups don't), there is a crapton of re-use when it comes to systems, and a classic webserver that used to be just for public stuff will just as much have some private applications added 'temporarily' (read: forever) and a crappy WAF / Proxy rule that is supposed to deny public access but gets bypassed with a simple URLEncode.
Doing the lower layers requires knowledge and dedication, of which the first is getting harder to find (not easier) and the second is getting squeezed out of most processes since it isn't something that gets quantified as value.
So no, it doesn't balance out, and no, the cloud doesn't do a magical new layer of things that on-prem couldn't do, even if on-prem usually fails to deliver on an abstraction layer (while the cloud does have it). A cloud does make it much more visible, cost-wise and impact-wise, because you can't hide in a cloud. What goes into a cloud API also comes out of the cloud API, there is no network scanning and hoping you find all hosts and appliances, everything that exists can be queried, and also gets billed with plenty of detail. On-prem has none of that, and the last 30 years of inventory/asset management attempts has proven that it's still something most on-prem setups don't do at all, or do a really crappy job at.
Re: Geico repatriates work from the cloud, continues ambitious infra overhaul
#60Earlier quoted context omitted.
The comment about "running legacy applications in the cloud was not any cheaper" stood out to me. Just moving the same legacy design into the cloud is not the optimal way to gain cost and availability improvements. If you have ever seen a data center from Azure, GCP or AWS, you will realize how difficult it will be for any company to compete in the long run. Those companies develop new generations of data center infr…
> They negotiate network and power contracts at a scale that exceeds any typical Fortune 500 company. ..and then mark it up. AWS overall has 38% operating margin[0]. Depending on your application this can hit you really hard (cloud egress bandwidth being an especially obscene offender). > I'm skeptical that running your own data center will end up a cost saver in the long run. It's not cloud -or- your own Azure-scale…
Speaking as someone who has seen (partially) behind the hyperscale curtain, I wholeheartedly agree.
Competing with an Azure, AWS, or GCP data center would absolutely be a _really_ expensive proposition, but it’s not something most Fortune 500s need (or want) to do. Hyperscaler data centers are intentionally designed to effectively be both available to (almost) every possible customer while also adhering to (almost) every GRC framework, redundancy metric, and security requirement that most of those potential customers may ask for.
If you’re running your own data center, you don’t need to worry about most of that. You only have to worry about your own needs or that of your customers.
The misconception that it’s either-or, or that the cloud is the prime solution for all use cases, is simply the result of really effective evangelism and marketing. That so many people working in software don’t have deep hardware expertise or are not familiar with data centers plays to that hand. Not a criticism, just an observation from my experiences.
Not that the cloud isn’t a very powerful option indeed.