Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geico repatriates work from the cloud, continues ambitious infra overhaul

thestack.technology

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#11

Post by x0x0 »

bluGill wrote: post #9
whatever1 wrote: post #2

If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.

If your data center isn't large enough to need at least 5 people full time admins then you should just go cloud. With a part time person you will see downtime when a machine fails. With 1 person that person will sometimes be on vacation when a zero day takes you down. With 2 people 1 will be on vacation when the second gets sick. You end up needing at least 5 people before you have enough people that you have redunda…

> With a part time person you will see downtime when a machine fails.

Many data centers offer remote hands services. And I don't believe this is at all true.

I worked at a place that managed thousands of boxes in dozens of pops with 1.5 fulltime people. If you design it for this from the beginning, with cattle not pets and netboot everywhere, this is very doable. And a large cost savings vs cloud.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#12

Post by mmcconnell1618 »

0xbadcafebee wrote: post #4

They had an expensive, fractured, hard to maintain on-prem layout. Then they moved to the cloud. And it turned out the cloud was expensive, fractured, and hard to maintain. So they're moving to on-prem. Any bets on what's going to happen next?

The comment about "running legacy applications in the cloud was not any cheaper" stood out to me. Just moving the same legacy design into the cloud is not the optimal way to gain cost and availability improvements.

If you have ever seen a data center from Azure, GCP or AWS, you will realize how difficult it will be for any company to compete in the long run. Those companies develop new generations of data center infrastructure with power efficiency improvements every single year. They negotiate network and power contracts at a scale that exceeds any typical Fortune 500 company. I'm skeptical that running your own data center will end up a cost saver in the long run.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#13

Post by miyuru »

0xbadcafebee wrote: post #4

They had an expensive, fractured, hard to maintain on-prem layout. Then they moved to the cloud. And it turned out the cloud was expensive, fractured, and hard to maintain. So they're moving to on-prem. Any bets on what's going to happen next?

according to the blog they started the cloud migration in 2013, there have been lot of improvements/changes to on-prem since then.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#14

Post by VirusNewbie »

whatever1 wrote: post #2

If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.

> expecting a significant ramp up of compute demand

Lots of data processing workloads don't need to be run constantly, but do need to be run in a shorter amount of time. Cloud is pretty good for that sort of thing.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#15

Post by geicosreyes »

I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment.

Geico's IT will slow to a crawl in the next years due to the immense madness of supporting Kubernetes on top of OpenStack on top of Kubernetes (yes, that's what they are doing).

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#16

Post by oneplane »

whatever1 wrote: post #10
oneplane wrote: post #6

Earlier quoted context omitted.

It's not really about cloud vs. on-prem, it's the fact that people cut corners and lack knowledge on-prem, and don't have the budgets to do anything about it. What you're referring to is mostly about elasticity, and it's true that if you don't need it, it doesn't make sense to pay for it. But that doesn't mean that on-prem (which almost always turns into a virtual machine shitshow with crappy network design -- which…

So when they are setting up config files for the cloud they don't cut corners? It is insane amount of work to follow safe practices to configure your cloud. I don't see that much difference compared to doing actual admin tasks.

The entire underlying layer of possible misconfigurations is absent in the cloud. Yes, the services on top of that can still be misconfigured, but you don't get access to hosts, SANs, switches, firewalls, gateways, there isn't anything for you to mess up. The shared responsibility model allows you to also pick even more robust options.

But even if you were to stick to something simple, say, object storage. A bucket or blob store has no SAN config, no webserver config, no switches, no gateways, no raid controllers, no striping, mirroring, parity configuration, no firmware, no BIOS, no BMC, no OS. None of that. It's all eliminated. All that remains is the top layer where you configure your cost-to-resilience ratio and your access policy. And yes, you could cut corners, but those are orders of magnitude fewer corners you could be cutting than if you include all the stuff below it.

Add to that: almost all of it has good APIs that are well defined, well supported and have an ecosystem to go with it. Try finding anything like that for a crappy NetApp or EMC appliance you find in a datacenter. It either doesn't exist, or it's so bad you might as well run MinIO or a bloody NFS share (not actual object storage) yourself.

Being bad at cloud is definitely more expensive than being bad at on-prem, I'll give you that. But with cloud, at least you get a bill that you can use to show your peers and higher ups that being bad has a cost. Internal virtual/amortised dollars are much harder to allocate to incompetence. It's often completely ignored, and at best revisited at periodic capacity planning reviews with few to no consequences.

The only place on-prem has, is with locality requirements. That includes latency sensitive things where sub 1ms is a goal, and air gapped things. But even in the first case things like an AWS Outpost exist, and those are cheaper than doing it yourself (not much, but enough to save on the hardware and on 2 FTEs).

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#17

Post by munk-a »

bluGill wrote: post #9
whatever1 wrote: post #2

If you don't have strong seasonality or not expecting a significant ramp up of compute demand (true for startups) why bother with the cloud? It is not more secure, I read every quarter about downtime events, and more importantly you have 0 control of your costs. Your company is likely not Amazon, you will do fine if you have your on prem computers.

If your data center isn't large enough to need at least 5 people full time admins then you should just go cloud. With a part time person you will see downtime when a machine fails. With 1 person that person will sometimes be on vacation when a zero day takes you down. With 2 people 1 will be on vacation when the second gets sick. You end up needing at least 5 people before you have enough people that you have redunda…

Additionally, as someone who has been a part of the interview process for IT people, if you only have two people and you're not an expert yourself there's a non-neglible chance that neither of the two people you've got are particularly good at their job. I'd advise any company to just accept the premium cost of using cloud services rather than risk getting ransomewared or what-have-you and finding out nobody ever actually tested the backups.

The cost of getting things wrong with on-prem aren't high on the average - but they sure are spikey if you get unlucky.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#18

Post by whatever1 »

oneplane wrote: post #16
whatever1 wrote: post #10

Earlier quoted context omitted.

So when they are setting up config files for the cloud they don't cut corners? It is insane amount of work to follow safe practices to configure your cloud. I don't see that much difference compared to doing actual admin tasks.

The entire underlying layer of possible misconfigurations is absent in the cloud. Yes, the services on top of that can still be misconfigured, but you don't get access to hosts, SANs, switches, firewalls, gateways, there isn't anything for you to mess up. The shared responsibility model allows you to also pick even more robust options. But even if you were to stick to something simple, say, object storage. A bucket o…

My friend some the biggest data leaks happened because of misconfigured S3 buckets which is literally one line of code to get right.

Cloud is not an insurance against incompetence.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#19

Post by stackskipton »

geicosreyes wrote: post #15

I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment. Geico's IT will slow to a crawl in the next years due to the immense madn…

This article read more like advertisement for VP spearheading all of this.

Re: Geico repatriates work from the cloud, continues ambitious infra overhaul

#20

Post by sofixa »

geicosreyes wrote: post #15

I've directly participated in this project and all I have to say is this: the same madness that created a super complex and unmanageable environment in the cloud is now in charge of creating a super easy and manageable environment on premises. The PoC had barely been approved and there was already legacy stuff in the new production environment. Geico's IT will slow to a crawl in the next years due to the immense madn…

> Kubernetes on top of OpenStack on top of Kubernetes (yes, that's what they are doing).

OpenStack's services are running in Kube? And Kube itself is ran as an OpenStack thing? Why? Why not use the same tooling used to deploy that initial Kube to deploy as many as needed? Still a massive maintenance burden, but you don't need to add OpenStack into the mix.

Post reply on HN