Live data from Hacker News

FedEx to close data centers, retire mainframes

datacenterdynamics.com

411–420 of 573 posts

Re: FedEx to close data centers, retire mainframes

#411

Earlier quoted context omitted.

I doubt they have too much COBOL lock-in, since the article notes their datacenter opened in 2008. Here's the thing: A mainframe may have spare capacity, but you still have to call IBM to unlock it. In order to make a Z run in a cost-effective manner, you need to run at 90%+ utilization at all times - which is excellent for batch jobs that can be scheduled, but is difficult to achieve with on-demand loads. You're pay…

> a phone call to big blue is not automated. Cloud autoscaling is. Respectfully, the hell you say. Cloud 'autoscaling' is something that has to be tenderly maintained by software engineers, who expect to be paid salaries and benefits and so on. It's not like FedEx can just rsync their data into the cloud and have all their software run forever. Instead, they need the engineering team that manages their existing data…

Why would one have to hire a whole new cadre of technical talent? Wouldn't it just be taking existing talent and/or ops teams and having them work with the autoscaling APIs as opposed to working with the data center teams. If the skillsets don't match then you can either train existing talent or, yes, hire new folks.

I agree the occasional phone call removed from your workflow isn't reason enough to justify a giant cloud migration... but it is one of many reasons.

Re: FedEx to close data centers, retire mainframes

#412
post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

That’s an incredibly silly take. Our whole economy is based on specialization. FedEx will never be as good at running datacenter as Amazon or Google. There’s room in the system for the cloud operator to earn a profit while still offering the product at a lower cost than anyone else can achieve in-house. Apple fairly notoriously hosts iCloud in other company’s clouds, and they’re a computer hardware and software compa…

The part about Apple is not accurate. Apple operates several dozen datacenters, has spent north of $30 billion building them and expanding them in recent years, operates their own Kubernetes (was Mesos) cloud platform that looks a lot like Heroku internally, and leverages public cloud for a couple parts of iCloud as a redundancy play, and no more. Maps helped prompt the expansion from single-homed iTunes legacy datacenter strategy, but even that iTunes legacy has been online since the early 2000s. Note that my context ends almost a decade ago, so, there’s that. The fleet dedicated to Maps alone when I worked there was large enough to be its own FAANG/MAMAA.

They’re in the datacenter game long term. In no universe does Apple “host iCloud in other company’s clouds”. Apple is notoriously a control freak and would never place their strategic services at the whim of cloud AZs. That idea was proposed and quickly eliminated. (Believe it or not, it is possible to spend that much on Azure/GCP and still only use it for basically blobs. How do you think they moved so easily in 2018ish?)

Re: FedEx to close data centers, retire mainframes

#413
post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

> Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. This "ultimate" endgame has been predicted since at least 2006 and we've yet to see anything but price decreases on cloud services. Tons of labor has been invested into deliberate cloud agnosticism with no apparent results. I…

To add to your point I think the competitiveness will increase over time with further technology innovation. Cloud providers are constantly adding features and trying to catch up with each other. New entrants are adding novel competitive dynamics as well (e.g. Cloudflare or Deno Deploy).

The cloud market is so large ($ TAM; also growing rapidly) I think there will always be a tailwind of investment and innovation that prevents monopolistic stagnation.

Re: FedEx to close data centers, retire mainframes

#414
post #23

Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money. If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point w…

Maybe this is true for public cloud pricing but it hasnt been true for some time when it comes to enterprises on cloud. Large enterprises come in and ask for a quote for X cpus/ram/storage over X years. They sign a multi year contract with a guaranteed minimum spend, and a discount on every line item. Then they "lift and shift" their physical data center into a cloud region, in a very static way (no autoscaling etc).

This works great for everyone, the hyperscalars have much lower costs than even the biggest enterprise customers so the company get a good deal. And the cloud company makes some revenue off the minimum spend (making capacity planning a lot easier) and they know once the compute and storage there it will be very likely the company starts using extra managed services.

Re: FedEx to close data centers, retire mainframes

#415

Earlier quoted context omitted.

You need to deal with overhead. Nobody does their own HVAC in house because you rarely need them, and would have to pay to train people on that despite them not using it. In some cases you can even get a discount. Utilities are. Big customer of tree trimming, the companies doing that work can give a great deal because the utility doesn't care that they take a week off after a storm for high profit margin consumer tri…

Lots of places have their own HVAC techs in house, if they have enough HVAC work to justify it. Even if it's not their core line of business. They will do whatever costs less, +/- some amount of subjective "hassle factor."

I have dealt with a large number of retail colo providers, wholesale data center providers and corporate owned data centers across the US over the last 20 years and all of them used contractors for HVAC and electrical. I'm not saying dedicated staff never happens but it is definitely not the norm.

Re: FedEx to close data centers, retire mainframes

#416

Earlier quoted context omitted.

You're paying for all of that whether it's the cloud or on-prem. The only real difference if you're a large company is whether you're paying another company's profit margin as well. If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space. There's a justifiable niche where you're too small to ju…

> benefit from the ability to rapidly scale on cloud platforms Big companies need this too. Some team wants to spin up some new service to try out some idea? In cloud-land they push a button. In "rack & stack" land they have to wait for Ops to purchase the hardware and provision it. Cloud makes it cheap and easy to try out new stuff.

Assuming you're doing dedicated machines for services. My company runs on-prem with a big cluster scheduler & maintains headroom in it; small deployments of new services and modest scale-ups of existing services don't require explicit capacity requests. Only if you're going to provision a huge number of instances do you need to wait for infra to buy machines. Which also requires advance planning with the "elastic" cloud anyway.

Re: FedEx to close data centers, retire mainframes

#417
post #111
post #23

Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money. If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point w…

It's not only about size, there's also workloads to take into consideration. Take FedEx, their operations are highly dynamic - per day but also per period. The number of packages sent, and being transported, vary greatly between a random Tuesday 15:00 and the weeks before holidays. In such a scenario, with your own infrastructure, you need to overprovision, by a lot, to be able to handle the heaviest load possible, a…

The flip side is the approach that Amazon took with AWS. Maintain the server capacity but invent some sort of mechanism to sell any extra server capacity during the down time.

Perhaps there isn't the mindset/ability to actually execute this though.

Re: FedEx to close data centers, retire mainframes

#418

Earlier quoted context omitted.

> Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? As long as they have backups, yep, that's a quite possible alternative. And backups cost very little, and don't need to run flawlessly 24x7. They will probably pay through their nose either way (backups or not, migrating or not), because that's how the cloud works. But well, that's for their bean-counters to count. As l…

Probably for the same reason nobody goes around talking about the BATNA of having employees. This doesn’t make any sense.

Not only people do talk about the BATNA of hiring people all the time, but I fail to see how this is relevant to a machine rental contract.

Re: FedEx to close data centers, retire mainframes

#419
post #36
post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational…

reminds me a bit of:

https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_...

"Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?"

from the post reply, in case it's not available from the URL:

alecsliu · 2 days ago · edited 2 days ago Gold2Eureka!Bravo!Today I Learned

Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal.

One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired.

So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually.

In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge.

This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant.

The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm.

The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow.

As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run.

Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha

Re: FedEx to close data centers, retire mainframes

#420
post #19

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure…

This line of reasoning has been around as long as the cloud has been a thing, and yet companies are still doing it and seeing better results than running their own data centers. At what point do we put this argument to bed? You want to talk about fear of switching costs? Fedex was still on mainframes, for crying out loud.
Post reply on HN