FedEx to close data centers, retire mainframes • Hacker News Board

Re: FedEx to close data centers, retire mainframes

#141

Post by toss1 » Tue, Jul 05, 2022, 12:42 PM UTC

oceanswave wrote: post #31

arghwhat wrote: post #24

Earlier quoted context omitted.

It certainly seems unlikely. Maybe it's the current operational cost, not subtracted the new operational cost. Maybe it's due to already adopting cloud and thus having paying double. Maybe it's due to the use of mainframes, rather than conventional servers. Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper…

So do you feel FedEx should make their own trucks, ships, and planes?

There is a huge difference between transport vehicles vs. data centers in terms of the amount of lock-in to particular vendors &cost of switching.

It looks to me like the cloud providers will soon have FedEx nicely tied over a barrel, and those 'savings' will prove illusory .

Re: FedEx to close data centers, retire mainframes

#142

Post by whatever1 » Tue, Jul 05, 2022, 12:43 PM UTC

sofixa wrote: post #111

hnarn wrote: post #23

Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money. If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point w…

It's not only about size, there's also workloads to take into consideration. Take FedEx, their operations are highly dynamic - per day but also per period. The number of packages sent, and being transported, vary greatly between a random Tuesday 15:00 and the weeks before holidays. In such a scenario, with your own infrastructure, you need to overprovision, by a lot, to be able to handle the heaviest load possible, a…

The cost does not scale linearly with the #of packages. A table with 1 entry is not cheaper than a table with 1 million entries.

The same routing algorithms have to run either they have one package or 1 million packages.

Plus you need to keep the history of the operations for months if not years (meaning that you will store both holiday and random Tuesday data anyway).

So where is this flexible cost exactly?

Re: FedEx to close data centers, retire mainframes

#143

Post by bad416f1f5a2 » Tue, Jul 05, 2022, 12:43 PM UTC

yarky wrote: post #124

usrusr wrote: post #46

Earlier quoted context omitted.

But that's only really a change when you come from commodity hardware. For those still on mainframes, what's the difference? Technically the hardware might be charged per replacement cycle instead of monthly, but vendors are just as empowered to demand whatever they want. It's certainly hard to avoid lock-in running stuff on a cloud provider, but I suspect that it is even harder when depending on ye olde mainframe.

I agree, isn't it way easier to switch vendors when running on the cloud? I'd say that encourages competition, since we can literally move the infrastructure using keystrokes instead of muscles. This makes me think computer infra could potentially become a commodity.

As always, it depends on how you use it.

Most cloud providers do let you run your own software on a managed substrate of infrastructure. But they also let you, nay, encourage you to build on top of their abstractions instead.

Re: FedEx to close data centers, retire mainframes

#144

Post by bluGill » Tue, Jul 05, 2022, 12:45 PM UTC

pid-1 wrote: post #130

cs702 wrote: post #36

Earlier quoted context omitted.

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational…

That seems to be a common take on HN - cloud is too expensive. I'm curious whether the folks claiming that have any data center ops experience. Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no…

To a company you need to pay those costs no matter what.

If the AC breaks at 3am it neds to be fixed. It doesn't matter if you have your own HVAC people on sight 24x7, your own people on call to service it, a local HVAC service to come in, or you outsource the entire operations and so you have no idea how that is handled. In the end the important part of this story is that whatever you are doing with the AC continues to work. Different operations demand different levels of service (I doubt that anyone keeps HVAC techs on staff 24x7, but if the AC is that critical it is mandatory). The only case where the CEO is up at 3am is if the CEO is the owner of the local HVAC service company, not the CEO of the building with the problem.

Once you realize that to management the cost is outsourced no matter what the only question is do it with your own people and HR, or outsource it. There are pros and cons to both approaches, but for most companies it isn't their business and so the only reason to do it in house is they can't trust any company they hire.

Re: FedEx to close data centers, retire mainframes

#145

Post by tyingq » Tue, Jul 05, 2022, 12:46 PM UTC

kragen wrote: post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

I do get your point, but if they have a fair amount of IBM mainframes, they are already renting their infrastructure. The mainframes have a culture of making you pay for operating system and other pieces on a basis of how much compute power you have "varied on". You don't really "own" a z-series box.

Re: FedEx to close data centers, retire mainframes

#146

Post by hef19898 » Tue, Jul 05, 2022, 12:52 PM UTC

oblio wrote: post #77

goindeep wrote: post #65

Earlier quoted context omitted.

I used to work with a very smart man that I'm sure was some kind of secret genius. He's was that sort of tech gofer. Hardware, software, didn't matter, if there was a problem he'd solve it. Sort of guy you'd see carrying a thick ass SQL book around because he 'needed to learn it' to solve just one little problem. He built whole entire solutions for the company I worked at in his spare time that the company once tried…

Never confuse movement for action :-)

As everybody who saw people, and orgs, rotating in place at incredible speeds can confirm.

Re: FedEx to close data centers, retire mainframes

#147

Post by seoaeu » Tue, Jul 05, 2022, 12:54 PM UTC

bluGill wrote: post #137

TheRealDunkirk wrote: post #98

Earlier quoted context omitted.

I'm 53, and I've worked for 3 Fortune 250's. Can confirm. I've seen this happen over and over again. Senior management makes some broad pronouncements, and the mid-level lieutenants have meetings with consultants, and then implement new, expensive projects that "will surely fix 'it' this time." Five to ten years later, after the dust settles, and we figure out that we've strapped YET ANOTHER LAYER of technical debt o…

8-10 years is as long as senior executives last, if they don't make a big change then thee is no way to take credit for their vision. Even if the company had a "perfect" org chart (as if such a thing is possible), they need to make changes otherwise someone will say that they old org chart is the cause of success and they as a leader were not worth anything. I don't know if the above fear would actually play out, nob…

I would expect it is just as much a hedge in case things go wrong. When your job is to steer the course of a company, it won’t look good if you crash and your hands weren’t even on the wheel

Re: FedEx to close data centers, retire mainframes

#148

Post by beckingz » Tue, Jul 05, 2022, 12:54 PM UTC

masswerk wrote: post #94

cs702 wrote: post #55

Earlier quoted context omitted.

Actually, I don't think the decision is that shortsighted in this case. It could work out pretty well for a decade or longer. IMHO FedEx is likely to have a good run before the costs and risks of this decision start outweighing its benefits.

However, when the situation starts to become unbearable, how long will it take to rebuild the infrastructure and source talent? Another decade? Adding the startup costs for any possible rebuild, you're apt for a substantial net loss. So you're probably going to shy away from such a reintegration, while losses accumulate further, until this becomes unsustainable, putting the entire corporation at risk. (The wise thing…

$400m saved today in exchange for $500m 10 years from now sounds like a good deal financially!

Sometimes tech debt can be used instead of financial debt!

Re: FedEx to close data centers, retire mainframes

#149

Post by sofixa » Tue, Jul 05, 2022, 12:55 PM UTC

throw0101a wrote: post #108

jon-wood wrote: post #95

Earlier quoted context omitted.

The immediate gains I can see are: * In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. * An extensive suite of complementary products are available from cloud providers, it's not just about being able to start a VM, in many cases y…

> In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. How many people run physical servers 'raw' anymore? I would hazard their current stack involves VMware, Hyper-V, Open Stack, or a combination of the three. You can have an API sys…

That handles IaaS, now do all databases (relational, NoSQL, NewSQL, KV, etc.), message brokers, object storage and a hundred other services. OpenStack handles some of those, with varying degrees of success, but with VMware and Hyper-V you have to DIY from scratch with IMHO the wrong level of abstraction.

Re: FedEx to close data centers, retire mainframes

#150

Post by IHLayman » Tue, Jul 05, 2022, 12:55 PM UTC

pid-1 wrote: post #130

cs702 wrote: post #36

Earlier quoted context omitted.

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational…

That seems to be a common take on HN - cloud is too expensive. I'm curious whether the folks claiming that have any data center ops experience. Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no…

ML workloads definitely cost a lot of money. Even for a preemptible VM, A100 GPUs cost $0.88/hr/GPU. That's $624 a month for a single GPU and only the 40GB model. Want a dedicated 8 GPU machine in the cloud to do training with? That'll run you around 16 grand a month. Do that for 2 years and you may as well have bought the device. Want to do 16/24/40 GPU training? Good luck getting dedicated cloud machines with networking fast enough between them so that MPI works correctly, and prepared to give up your wallet.

Also, that's just compute. What about data? Sure cloud accepts your data cheaply, but they also charge you for egress of that data. Yes you should have your data in more than one location, but if you depend on just cloud then you need it in different AZ which costs even more money to keep in sync and available for training runs.

I think for simple workloads and renting compute for a startup, cloud definitely makes sense. But the moment you try to do some serious compute for ML workloads, good luck and hope you have deep pockets.