Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bank of America's CEO says it's saved $2B per year by building its own cloud

businessinsider.com

Post reply on HN
Print view View original

291–300 of 321 posts

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#291

Post by bonesss »

m463 wrote: post #232
jmartrican wrote: post #154

Earlier quoted context omitted.

I worked on the internal BOA cloud. My team wrote software to procure VM's via a website, versus the older (still current) way of doing it manually. There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily…

Maybe it can make sense for a small company trying to scale where time is a factor using aws. But I would expect that a large established company would do better to create it's own reasonably priced reasonably managed infrastructure. Sort of rent vs build/own.

That's generally the economic case put out in cloud planning: a startup or expansion does great in the cloud with quick time to market, but once that business has matured and the risk is gone the assets move in-house because it's wildly cheaper.

And something that seemingly everyone forgets: it's not black and white... you can have some things in the cloud while you have some thing on-prem, leveraging scalable infrastructure and redundancy while also keeping your core assets fully under your control. It's pretty easy to make an internal S3 replacement, it's hard to make a better S3.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#292

Post by bonesss »

kthejoker2 wrote: post #283
dodobirdlord wrote: post #272

Earlier quoted context omitted.

> What's worse, someone being a prima donna or the unchangeable facts of AWS operation policies? There's this great thing about BOA-sized support contracts where you get to call up the senior product managers and demand that they build the features that you want. And if you're not happy about something, engineers will drop what they're working on to appease you. I wonder how many customers were interested in setting…

Can confirm, I'd say 20-30% of all Azure features come directly from the product teams' Partner Advisory Council (read: top customers) meetings. Like "snap your fingers" implementation.

And what's the alternative? Losing your biggest customers because you couldn't be assed to put a new button on some load-balancer configuration?

Of course, for small or medium customers whose finger snaps are ignored... WYSIWYG. The roadblocks in not-your cloud are still your roadblocks.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#293

Post by KaiserPro »

Jestar342 wrote: post #192
KaiserPro wrote: post #173

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

> the opex was close to £3mil in AWS bills alone What was the opex before cloud?

Thats difficult to work out. Pure hosting was easily less than £1M

But software licensing costs play a massive factor in the overall opex.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#294

Post by KaiserPro »

tapirl wrote: post #237
KaiserPro wrote: post #173

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

It is also more expensive for very small companies and individual developers. You need pay extra for inbound/outbound traffic and any persistent disk space to cloud providers, whereas these are all included in the plans provided by most VPS providers.

Its not all that clear cut.

If you are paying significant cash for network, then what ever you are doing you need a CDN of some sort (I assume most people are web here.)

VPS bandwidth is not unlimited, and it is a single point of failure.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#295

Post by KaiserPro »

rosswilson wrote: post #182
cpeterso wrote: post #176

Earlier quoted context omitted.

> mixture of SUN and Intel blades all backed by FC What is FC?

Fibre Channel, I think: https://en.m.wikipedia.org/wiki/Fibre_Channel

Sorry, yes, Its fibre channel.

There was a full rack switch, which must have been easily 400 + ports.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#296

Post by bonesss »

mumblemumble wrote: post #218
KaiserPro wrote: post #173

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

I used to work at a very large and entrenched publishing-cum-tech company, too. Sounds like they were different companies, but very much the same rationale for migrating to a well-known cloud service provider. The ossified and entrenched IT department was becoming one of the biggest challenges to quickly reacting to market challenges and opportunities, and was long overdue for a few boots planted firmly to the rear e…

Another way of framing that: BOFHs went from being necessary to the business to preventing the business from acheiving its goals, and got replaced by competition that was more amenable at a higher price.

They weren't seen as so unprofitable they had to be eliminated, but unprofitable enough that competition could triangulate their price vs pain.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#297

Post by KaiserPro »

weberc2 wrote: post #206
KaiserPro wrote: post #173

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

> It will continue to rise because it offers flexibility ... The cloud is more expensive for large companies, for most workloads. The way I read this is that the operating costs are higher in the cloud, but presumably dwarfed by recouped opportunity costs. This makes sense to me, given my limited experience working in large companies where every interaction with IT involved a 2-6 week turnaround time. Need a VM? Fill…

Pretty much.

Its not so much the availability of physical servers (although that might be an issue) it was the features, and the time it took to port new bits and bobs.

A classic example is databases. We had a puppet script that provided a HA-mysql cluster. However, it was very fragile, never really worked and I'm mostly sure the backups never actually took place.

It would take me (a devop who was tasked with doing this sort of thing) the best part of 3 days to setup and verify. The whole thing collapsed when the DBA that made it left.

RDS is a one click operation. yes its more expensive to run long term, but the saving on staff time was enormous. I will never go back to inhouse rolled puppet, unless it was proven to be outstanding.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#298

Post by vinay_ys »

mfer wrote: post #35

> Right now, the bank estimates its private cloud is 25% to 30% cheaper than public providers, though it also recognizes that probably won't last forever. I am glad they shared this situation. We often here marketing telling us that public cloud is always better. It's nice to see different perspectives being shared and some details around them.

If you have $50million+ you can deploy on capex in one shot (economies of scale negotiation doesn't work well at lower numbers), and you can hire systems engineers (C/C++/Go/Rust 5+ years systems experience x 20 engineers), you can put together a single tenant compute cloud needed to run a typical Internet facing web/mobile application with all the bells and whistles (lots of app server clusters, db clusters, big data clusters, GPU clusters etc with HA, scalability, security, DR etc). At this point, capex/opex cost will be easily 30% lower than the best negotiated bulk price from public cloud.

It is worth doing this if your in-house engineering team is at least 100+ app service developers and have high feature churn rate.

But be warned, your systems engineering / shared technology team should be level-headed and mature and rest of your app service engineering should be good too to pull this off well. If not, you will be in serious developer productivity pain and there won't be quick and easy fixes once you have put down the capex.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#299

Post by weberc2 »

greglindahl wrote: post #269
weberc2 wrote: post #261

Earlier quoted context omitted.

No, it wouldn’t make sense for me to make the 95/5 distinction only to prescribe a 100/0 response. My post was a call for sober judgment, which is admittedly not very popular here.

Making up opinions and then arguing against them is not popular here.

No straw men here Greg. Looks like you’ve confused this thread with another.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#300

Post by bonesss »

davnicwil wrote: post #194
gumby wrote: post #24

I'm not really sure why this should be a surprise. Just as big companies use a mixture of owned and rented real estate for various reasons, the same is true of other large expenses. If you have a large, predictable, core workload it makes sense to bring it in house, and use the elastic (rented) resources for unpredictable stuff (e.g. new efforts). It's not like Amazon doesn't know this too (e.g. they try to make the…

Agreed - past a certain (probably fairly massive) threshold of predictable long term load, it seems clear renting resources is going to be more expensive than the all-in cost of owning and operating your own. Were the opposite true, you'd wonder about the economics of the cloud business model. I think a more interesting way to look at this is, are the other advantages of cloud services (e.g. lots more flexibility and…

> but does that cost you the opportunity to make even more than that, given you're probably moving slower or at least less efficiently than you otherwise could?

The potential opportunity of fast-moving cloud features needs to be weighed against the opportunity costs of slow-moving cloud features. Where bespoke solutions can immediately provide tailored performance and maximize technical capabilities, a missing feature in any of your cloud providers services can be a showstopper or unmitigatable roadblock. And while lots of the technology is past the bounds of reasonable economical replacement, some of the technologies being shared through the cloud are nigh unfathomable to recreate.

Which is to say that the black ju-ju behind Windows update probably takes making a new MS to build up to present maturity, and unless you're a certified "big boy" letting small teams somewhere else fully dictate what you can and can't do at service boundaries probably impacts you in the long run.

Based on that, and IMO/IME: the answer isn't a binary choice but a constantly shifting point on a spectrum between the two, where on-premise/local-cloud and remote-cloud services are aware of one another and maximize capabilities while minimizing costs. Hybrid installations are just stronger, and are easier to reshape according to costs.

Post reply on HN