Live data from Hacker News

Geico repatriates work from the cloud, continues ambitious infra overhaul

thestack.technology

31–40 of 63 posts

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#31
post #6

If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.

It's not really about cloud vs. on-prem, it's the fact that people cut corners and lack knowledge on-prem, and don't have the budgets to do anything about it. What you're referring to is mostly about elasticity, and it's true that if you don't need it, it doesn't make sense to pay for it. But that doesn't mean that on-prem (which almost always turns into a virtual machine shitshow with crappy network design -- which…

That's really what some/most companies want, a platform that can run cheap, fast and easy VMs, like on-prem, but without the hassle of having to deal with the hardware and physical network part, like in the cloud. Sadly that's not the choice being offered.

I don't know, I've seen the shittiest stuff built on-prem and in cloud, and I've seem completely amazing on-prem infrastructure and cloud stuff that could not possibly be built outside AWS.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#32
> In an interview with The Stack she confirmed the shift, saying “we have a lot of data – and it turns out that storage in the cloud is one of the most expensive things you can do in the cloud, followed by AI in the cloud…”

This has been the story for 20 years now. Not even exaggerating. We all knew it was expensive from the get-go because we all did things on prem.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#33

They had an expensive, fractured, hard to maintain on-prem layout. Then they moved to the cloud. And it turned out the cloud was expensive, fractured, and hard to maintain. So they're moving to on-prem. Any bets on what's going to happen next?

> Any bets on what's going to happen next?

Someone in the c-suite gets a massive bonus before moving to a new company.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#34

They had an expensive, fractured, hard to maintain on-prem layout. Then they moved to the cloud. And it turned out the cloud was expensive, fractured, and hard to maintain. So they're moving to on-prem. Any bets on what's going to happen next?

The comment about "running legacy applications in the cloud was not any cheaper" stood out to me. Just moving the same legacy design into the cloud is not the optimal way to gain cost and availability improvements. If you have ever seen a data center from Azure, GCP or AWS, you will realize how difficult it will be for any company to compete in the long run. Those companies develop new generations of data center infr…

> They negotiate network and power contracts at a scale that exceeds any typical Fortune 500 company.

..and then mark it up. AWS overall has 38% operating margin[0]. Depending on your application this can hit you really hard (cloud egress bandwidth being an especially obscene offender).

> I'm skeptical that running your own data center will end up a cost saver in the long run.

It's not cloud -or- your own Azure-scale datacenter. There are any number of approaches in between including hybrid to offload stuff like CDN, storage, edge services, etc to cloud but the fact remains many companies can run the entire business from a few beefy machines in co-location facilities. Most companies, solutions, etc are not actually Google, Snapchat, Geico, etc scale and never will be.

Throw in some minor accounting tricks like leasing (with or without Section 179) and these kinds of "creative" approaches are often impossible to beat from a pricing/performance and even uptime standpoint. That's certainly been my experience.

[0] - https://www.theinformation.com/articles/why-aws-fat-margins-...

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#35
post #9

Earlier quoted context omitted.

If your data center isn't large enough to need at least 5 people full time admins then you should just go cloud. With a part time person you will see downtime when a machine fails. With 1 person that person will sometimes be on vacation when a zero day takes you down. With 2 people 1 will be on vacation when the second gets sick. You end up needing at least 5 people before you have enough people that you have redunda…

> With a part time person you will see downtime when a machine fails If a hardware failure causes downtime you're doing it wrong. Additionally, big cloud scaring people from hardware with marketing and FUD has been very effective. Modern hardware is insanely reliable and performant - I don't think I've seen a datacenter/enterprise NVMe drive fail yet. It's not 2005 with spinning disks and power supplies blowing up le…

To me, the downside of on premise hardware isn't hardware swap out, it's just dealing with hardware in general. All hardware needs updates which is downtime for that hardware. Also, anyone in this industry long enough has been around for "Oh, we will just replace that broken piece of hardware" that ended up "WHY IS EVERYTHING ON FIRE?" because versions didn't match up, hardware was rejected or just plain "Actually, THAT failure mode isn't redundant."

That can happen to Public Cloud as well but since they work with hardware at much much larger scale and most of time, build actual hardware software, they are much more aware of sharp edges.

Finally, with Broadcom acquisition, what virtualization software are using and is it really cheaper then the cloud?

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#37

I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment. Geico's IT will slow to a crawl in the next years due to the immense madn…

First you charge them to put a star on their belly, and then you can charge them to take the star off their belly!

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#38

If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.

Because you're not Amazon you also probably don't have tech as your core competency and don't have the budget to hire people skilled enough to operate an your on-prem setup as well as they operate their cloud.

Because you're not a startup you there is a very good chance that you have a very process-driven (cover your ass), slow-moving culture - this very often translates to an IT department where getting even basic things done (like reserving extra compute or changing a network setting or starting to use a third party software) takes months of waiting or pleading. Maybe you have never encountered this kind of pathological IT department, but they're very common, and it's a major reason executives bought into cloud to begin with. Of course, many companies like Geico seem to have merely replicated their IT pathologies in the cloud, but at least in the cloud you have fewer sources of problems in areas like physical space management, buying/integrating hardware to grow or change your footprint and dealing with all the SKUs and supply chain problems therein, or negotiating on-prem licences.

There are many more moving pieces when operating on-prem: more operations staff across more kinds of roles (yes, you still have eg devops people when using the cloud, but you don't need as many building operations staff (where managing a datacenter is its own speciality), people managing hardware/software vendors and related supply chain issues, people skilled in physical networking, people to plug things in/out and physically operate the machines), managing and acquiring the physical space where your on-prem setup is, buying/accounting for all the different kinds of hardware you need, licensing/using more software with more difficult integration to achieve equivalent functionality to eg EC2, licensing all your 3P software to run on-prem... even if nominally less expensive than the cloud in some cases, there are many more places where things can go wrong. That's not as easy to account for in a direct TCO comparison because it manifests as slowing things down - which does introduce very substantial costs - and distracting management away from other opportunities to grow revenue or improve costs.

Also, cloud downtime is really overstated as a problem in 2024. It makes the news because it has a high blast radius and involves high profile companies, not because it's more common than on-prem. With the exception of AWS us-east1 issues (which can break many AWS products at once across the world), most cloud reliability issues these days are isolated to only a few products and only a few regions. I think a lot of small on-prem companies don't realize that they are not actually more reliable, but just operate at a smaller scale where the probability of downtime causes "lucky streaks" to be more common (ie if you play roulette for three rounds, you're much more likely to have an abnormally high win rate than someone who plays it for three hundred rounds, even though you both have the same odds). Most companies don't have as mature security/risk operations as cloud providers and so face an existential risk/the possibility of huge (months) of downtime in the event of a fire/natural disaster at their dc, cryptolocker attack, janitor unplugging the server that says "do not unplug" - this isn't something people have to worry about with cloud providers to nearly the same extent.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#39

Earlier quoted context omitted.

> With a part time person you will see downtime when a machine fails If a hardware failure causes downtime you're doing it wrong. Additionally, big cloud scaring people from hardware with marketing and FUD has been very effective. Modern hardware is insanely reliable and performant - I don't think I've seen a datacenter/enterprise NVMe drive fail yet. It's not 2005 with spinning disks and power supplies blowing up le…

To me, the downside of on premise hardware isn't hardware swap out, it's just dealing with hardware in general. All hardware needs updates which is downtime for that hardware. Also, anyone in this industry long enough has been around for "Oh, we will just replace that broken piece of hardware" that ended up "WHY IS EVERYTHING ON FIRE?" because versions didn't match up, hardware was rejected or just plain "Actually, T…

> Also, anyone in this industry long enough has been around for "Oh, we will just replace that broken piece of hardware" that ended up "WHY IS EVERYTHING ON FIRE?" because versions didn't match up, hardware was rejected

I've been doing this for 25 years and I'm not sure what this means. Dell isn't going to come back to you and say "sorry but we can't fix this". With the warranty SLA worst case scenario they'll just replace the entire machine if they have to although I don't remember ever seeing it come to that.

> just plain "Actually, THAT failure mode isn't redundant."

When it comes down to it similar issues exist with clouds - regions, availability zones, etc. Big clouds have had multiple widespread outages just this year[0].

From that reference you can see that MS and Amazon themselves struggle to design, build, and run solutions for their own products in their own clouds.

It's always interesting to see marquee household name companies/products/solutions go down when US-East (or whatever) is having a bad day again.

Cloud can be a lot of things but a silver bullet for reliability and uptime isn't one of them.

[0] - https://www.forbes.com/sites/emilsayegh/2024/07/31/microsoft...

Post reply on HN