Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FedEx to close data centers, retire mainframes

datacenterdynamics.com

Post reply on HN
Print view View original

371–380 of 573 posts

Re: FedEx to close data centers, retire mainframes

#371

Post by ElectricalUnion »

hoppyhoppy2 wrote: post #248
IHLayman wrote: post #150

Earlier quoted context omitted.

ML workloads definitely cost a lot of money. Even for a preemptible VM, A100 GPUs cost $0.88/hr/GPU. That's $624 a month for a single GPU and only the 40GB model. Want a dedicated 8 GPU machine in the cloud to do training with? That'll run you around 16 grand a month. Do that for 2 years and you may as well have bought the device. Want to do 16/24/40 GPU training? Good luck getting dedicated cloud machines with netwo…

It's probably worth remembering that a company the size of FedEx isn't going to be paying the listed prices.

They actually probably be paying more that the average listed prices, as Mainframe (basically a on-premise PaaS, where IBM rents you high performance, distributed and redundant hardware cluster on a pay-as-you-use manner) users are often dependent on very high reliability, high uptime and low latency.

Re: FedEx to close data centers, retire mainframes

#372

Post by wnevets »

kragen wrote: post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

You're forgetting the most important part of this entire thing, the massive bonuses the current executives are going to get for saving the company over $400m. Whatever happens to the company afterwards is not their problem, they're already moved on to the next company to spread the legend of them saving FedEx over $400m.

Re: FedEx to close data centers, retire mainframes

#373

Post by plonk »

SoftTalker wrote: post #265
RandomWorker wrote: post #216

Earlier quoted context omitted.

right now, they can't retain the people or afford the people to do this work. When you can go work elsewhere for more money you move. And, running a main frame isn't just having it, it's keeping the people running it, plus paying for the electricity and space. Depending where, the real-estate and energy prices are nuts in most places. And, the engineers are expensive right now, and the services are cheaper. It's not…

> These managers see the larger trends, rising energy costs, maintenance costs rising, hiring difficult, and retention of existing engineers impossible. But how will the cloud providers avoid these trends? They won't. They will have to do the same thing as anyone else: pay more. And therefore charge more. There are economies of scale, but those savings are logarithmic and a company like FedEx is already pretty far ou…

> a company like FedEx is already pretty far out on the X axis

AWS and Azure probably run hundreds of Fedex-sized clouds and it's their core business. I don't think Fedex is very far compared to them.

Re: FedEx to close data centers, retire mainframes

#374

Post by jodrellblank »

SoftTalker wrote: post #265
RandomWorker wrote: post #216

Earlier quoted context omitted.

right now, they can't retain the people or afford the people to do this work. When you can go work elsewhere for more money you move. And, running a main frame isn't just having it, it's keeping the people running it, plus paying for the electricity and space. Depending where, the real-estate and energy prices are nuts in most places. And, the engineers are expensive right now, and the services are cheaper. It's not…

> These managers see the larger trends, rising energy costs, maintenance costs rising, hiring difficult, and retention of existing engineers impossible. But how will the cloud providers avoid these trends? They won't. They will have to do the same thing as anyone else: pay more. And therefore charge more. There are economies of scale, but those savings are logarithmic and a company like FedEx is already pretty far ou…

They aren't playing the same game. Facebook designed its own servers, they were chassis-less, didn't have a mains power input so no switch-mode power supplies, instead they had a 12V DC feed, they had no rack-wide large UPS instead each server had a small battery in it, they were not built for massive redundancy like a Dell server with dual PSUs and redunant networking, because they were disposable nodes in a larger software cluster, e.g. [1] [2]

Things that aren't Fedex's core competency.

Or, Microsoft's roofless datacenters[3], or locating datacenters in remote and colder climates, things the big players can do with economies of scale beyond buying things cheaply, they can customise the entire datacenter. Microsoft has experimented with underwater datacenters[4] and modular containerised datacenter extensions[5] which could be datacenters no human needs to be near to work on, or which could be dropped off somewhere with cheap land and power and internet, and picked up three years later and retired from use, or etc. Ideas which are not FedEx's core competency and need large scale and software clustering on top.

While FedEx would be hiring ordinary IT employees to work in a standard datacenter in cheap business park - not very enticing - Amazon could be hiring datacenter workers to work with Amazon's undersea cabling connecting their worldwide datacenters; more enticing work for skilled employees.

Google has been known/rumoured to migrate heavy batch processing workloads around the planet, following the day/night cycles to take advantage of regional cheaper night-rate electricity all the time. Something which reduces their energy costs but which FedEx may not be big enough to do.

[1] https://engineering.fb.com/2016/03/09/data-center-engineerin...

[2] https://engineering.fb.com/2019/03/14/data-center-engineerin...

[3] http://www.500eco.com/exhibits/microsoft-roofless-datacenter...

[4] https://news.microsoft.com/innovation-stories/project-natick...

[5] https://www.datacenterdynamics.com/en/news/microsoft-develop...

Re: FedEx to close data centers, retire mainframes

#375

Post by spookthesunset »

ocdtrekkie wrote: post #186
mbesto wrote: post #172

Earlier quoted context omitted.

> rent all information infrastructure Most of them were renting most of the infra anyway: - Co-location (physical space rented) - Managed service to keep the lights on (power, back-up generators) - Leased hardware that gets replaced every 3 years - Managed service for switch, firewall, etc. monitoring - ISP, back-up generators, etc. > but that will likely be many, many years down the road They're going to explain to…

You're paying for all of that whether it's the cloud or on-prem. The only real difference if you're a large company is whether you're paying another company's profit margin as well. If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space. There's a justifiable niche where you're too small to ju…

> benefit from the ability to rapidly scale on cloud platforms

Big companies need this too. Some team wants to spin up some new service to try out some idea? In cloud-land they push a button. In "rack & stack" land they have to wait for Ops to purchase the hardware and provision it.

Cloud makes it cheap and easy to try out new stuff.

Re: FedEx to close data centers, retire mainframes

#376

Post by jotterbrain »

jeffbee wrote: post #233
nix23 wrote: post #223

Earlier quoted context omitted.

Why do you buy servers with metal flakes in it? No quality controll on your side?

Are you saying that part of the expected savings from going on-prem is that you will have to disassemble equipment bought from major OEMs and examine it for microscopic metal dust? That doesn't sound like it will save much money, honestly.

They’re saying it’s a surprise to hear that Dropbox doesn’t know what QC and order acceptance means. And it is, I agree. That you spent the time investigating it, implying those servers were in production, is a shibboleth to those of us that know what we’re doing when designing hardware usage that Dropbox doesn’t. It is, however, your self sourced report and we don’t have an idea of scale, so maybe they do and you’re just unlucky.

And no, operators don’t disassemble to perform QC. And no, I could hire an entire division of people buying servers at Best Buy, and disassembling them, and stress testing them, and all of that overhead including the fuel to drive to the store would still clock in under cloud’s profit margin depending on what you’re doing.

You’re of course entitled to develop your cloud opinion from that experience. That’s like finding a stain in a new car and swearing off internal combustion as a useful technology, though, without any awareness of how often new cars are defective.

Re: FedEx to close data centers, retire mainframes

#377

Post by systemvoltage »

cs702 wrote: post #36
kragen wrote: post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational…

Some are starting to question the almighty Cloud™: https://www.economist.com/business/2021/07/03/do-the-costs-o...

One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down.

I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.

Re: FedEx to close data centers, retire mainframes

#378

Post by scottyah »

masswerk wrote: post #159
beckingz wrote: post #148

Earlier quoted context omitted.

$400m saved today in exchange for $500m 10 years from now sounds like a good deal financially! Sometimes tech debt can be used instead of financial debt!

Mind that if you'd ever wanted to integrate again, this means buying land, planning infrastructure, building it, planning, obtaining and setting up hardware and software, hiring, training, defining and testing procedures, etc, as you're starting from zero again – and all this while the costs, which forced you to consider this move, are piling up. Odds are, you'll never do this and cloud providers will know this. The…

FedEx had very few on site people managing the hardware, it seemed like most work on hardware was done by the suppliers. There were also very few servers there and actually running, I think the software powering the enterprise took up a lot less than they expected.

Either way, I think you're spot on with the "training" part. FedEx was one of the first companies to raise significant capital and pursue a business dependent on technology. It also treated its people very well so relatively few left since the 70's. I think they just didn't train the next generation enough and they realized that those skills are disappearing rapidly because no college grads want to learn old & boring stuff and everyone who does know it costs a lot to keep (FedEx still heavily relied on COBOL when I left ~5yrs ago).

Re: FedEx to close data centers, retire mainframes

#379

Post by iamtheworstdev »

goindeep wrote: post #65
cs702 wrote: post #36

Earlier quoted context omitted.

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational…

I used to work with a very smart man that I'm sure was some kind of secret genius. He's was that sort of tech gofer. Hardware, software, didn't matter, if there was a problem he'd solve it. Sort of guy you'd see carrying a thick ass SQL book around because he 'needed to learn it' to solve just one little problem. He built whole entire solutions for the company I worked at in his spare time that the company once tried…

there used to be an allegory of how to identify a bad, new CIO... if your current system was massive, networked printers that were shared amongst floors.. the new CIO would come in and give everyone individual printers... or vice versa .. because 'change'

Re: FedEx to close data centers, retire mainframes

#380

Post by citizenpaul »

Shorel wrote: post #365

Not saving, just giving those $400m to AWS or Azure, over some amount of time =)

Long enough for this exec team to "prove" on paper they saved money during their tenure. Than hit the road and the next exec team to move it back because 'ooops'
Post reply on HN