Live data from Hacker News

Shortage of COBOL developers in insurance and banks

lemonde.fr

51–60 of 72 posts

Re: Shortage of COBOL developers in insurance and banks

#51
post #40

Earlier quoted context omitted.

> Almost all of this seems untrue (except maybe the last one). I’ve worked at a variety of banks from front to back office across securities and private wealth. I’ve seen plenty of company-specific languages, plenty of C++, Java, perl, python but have literally never encountered a COBOL system. Two of Australia's four biggest banks – ANZ and Westpac – have historically run CSC Hogan. I don't know what they are doing…

They did good by not switching to the Oracle Banking Platform, after all, it's Oracle.

Like many Oracle products, they didn't write it from scratch, they acquired it – from Citibank. In the early 1990s, Citibank set up a subsidiary in Mumbai to maintain their banking software, and sell it to other banks. Citibank ended up selling that subsidiary to Oracle, although Oracle is only the majority shareholder, and a minority of the stock is listed on the Bombay Stock Exchange.

When I worked at Oracle, I was involved on the periphery of it, in the middleware products that it used, but I never actually saw the product itself.

Re: Shortage of COBOL developers in insurance and banks

#52

Earlier quoted context omitted.

Oh that's not the case here, management would love to get off these systems. They see the risks with the age profile of developers, staffing issues, and general ageing tech problems. Re-writing these systems in another language is almost not an option. They have been written over decades with countless undocumented projects. They are core systems that work the way the business wants (or is built around, for better or…

Gotta rip the bandaid somehow. Even if it's not perfect, at least it would be more maintainable than COBOL.

[deleted]

Re: Shortage of COBOL developers in insurance and banks

#53

Earlier quoted context omitted.

Oh that's not the case here, management would love to get off these systems. They see the risks with the age profile of developers, staffing issues, and general ageing tech problems. Re-writing these systems in another language is almost not an option. They have been written over decades with countless undocumented projects. They are core systems that work the way the business wants (or is built around, for better or…

Gotta rip the bandaid somehow. Even if it's not perfect, at least it would be more maintainable than COBOL.

Ripping the bandaid could potentially cause fatal bleeding. Keep in mind these stacks are calcified on a lower layer of stack and other higher ones depend on.

Re: Shortage of COBOL developers in insurance and banks

#54
post #5

As many said before ; it’s not learning Cobol (or Fortran), it’s the weird IBM mainframe/OS ecosystem that is the problem. I am good at both languages, but I know very little about the ecosystem it runs in.

As a Millennial, I haven't ever seen/touched/used a mainframe. And I work in a modern Data Center. What are the pros/cons of mainframe? What makes them better than a rack housed with necessary hardware? Well, I can see cons - not so accessible as regular hardware. Impractical/impossible to have at house. So the learning curve for hobbyists/tinkerers/curious are missing first steps.

My simplified understanding is that mainframes take vertical OLTP scaling AND system redundancy to the absolute extreme.

In any other computer your small number of general purpose CPUs have to handle all the business / application logic plus disk IO, network IO, cryptography, system housekeeping, etc. On a mainframe all those other things are offloaded to specialized coprocessors and the actual business logic gets nearly 100% of the cycles on hundreds of special purpose high frequency water cooled business logic CPUs running at 5+GHz.

Also any component can fail and be replaced without downtime or lost transactions. CPUs, RAM chips, entire motherboards, anything at all. You can rebuild it online while your transactions continue. Transactions can be run in duplicate across multiple hardware paths to ensure they are never lost and also verify output correctness.

If you set out to build a database server that handles 100 million transactions per hour and never ever goes down, you would end up with a mainframe.

Re: Shortage of COBOL developers in insurance and banks

#55

I would do it for them for the right price; about 5x what I currently make writing Java/Js/C on Linux. These sense like an extreme amount, but the problem is I’m pigeon holing myself into a single tech and my pathway to career advancement is finding other big banks that are hurting… which again I’m willing to do, but the risk/reward has to be there. Given what some of these mangers make at these banks, coughing up th…

> Given what some of these mangers make at these banks, coughing up that salary isn’t unreasonable.

From the bank's perspective, it is unreasonable.

Legacy banks still don't understand/accept that IT is the foundation of their business. Maybe a century ago it was possible to run a bank on paper records, typewriters and abacuses (abacii?) but those days are long gone. Yet, the banks still consider IT as a nuisance and overhead to be outsourced to the lowest bidder (and many consultancies have third-world sweatshops full of incompetent monkeys to fill this niche). It is also a positive feedback loop because outsourcing IT to idiots means the user experience is terrible and the people within the bank now have a good reason that reinforces that mindset.

Your typical software engineer/IT person in the bank has about the same clout and political power as the janitor, absolutely not enough to commend the high salaries given out to managers and various paper-pushers. Keep in mind it's still a very nepotistic environment (relative to most tech companies) and in a lot of cases those managers don't get those high salaries upfront but grew them over time by playing office politics correctly and doing the right favors to the right people.

Re: Shortage of COBOL developers in insurance and banks

#56
post #5

As many said before ; it’s not learning Cobol (or Fortran), it’s the weird IBM mainframe/OS ecosystem that is the problem. I am good at both languages, but I know very little about the ecosystem it runs in.

As a Millennial, I haven't ever seen/touched/used a mainframe. And I work in a modern Data Center. What are the pros/cons of mainframe? What makes them better than a rack housed with necessary hardware? Well, I can see cons - not so accessible as regular hardware. Impractical/impossible to have at house. So the learning curve for hobbyists/tinkerers/curious are missing first steps.

Depending on your starting level they are designed in a way which is ahead of time for many small software houses environments even now.

I went through couple of mainframe z/OS basics certifications about 10y ago, never did anything commercial, just for educational purposes. All info comes from lectures and people who visited the cert-learning group with job offers.

A bit of history, mainframes are done by IBM which historically had some interesting things to do like count votes which led them into data processing market, mainframes were built for high data loads. You don't buy mainframe, you only rent it, physically. The level of duplication possible with these machines is incredible. You might receive a call, or maybe mail nowadays that something has broke in the mainframe you are using and you might be not aware it ever happened. I don't know if any "regular" vendor can tell you that e.g. half of 24 CPU board is broken down and they are going to replace it for you as this is their responsibility and would like to be sure someone can open the building with mainframe. Pro tip: you might be scared because you know you need 40 CPUs and that means you should have 2 such CPU boards and now you think you are missing 4 CPUs, but in reality you will learn that it's all ok and you will not face downtime because they can change it without switching it off and anyway you already had 3 boards just in case some failure happens and IBM does not want to send technicians back and forth multiple times a year for something as simple as this thing so they just have put more parts into your chassis and since telemetry is working all the time the know when they have to replace parts. On the software side you can run pretty much anything fancy there, the can emulate whatever you need if only enough clients have enough $$$ and this is how they can offer you CPUs with native java support, you don't run assembly there, you run JVM bytecode directly on CPU. And the more accurate software side, sometimes you have to deal with strange limitations, are they wrong? I don't know for sure, but I rather think not wrong. Mainframes can operate in backward compatible way and you are hostage to thinking of times when computers very in early evolving stage and world looked different. If you are writing web apps - which is pretty much kind of majority of work for devs in my experience - have you ever considered things like limits of the app you are writing? Can you estimate a number of users who can use app concurrently without measuring? Do you know upfront how much space for logs do you need? Maybe yes, but if you have to tell your operating system that you need space for 5 billions of log records and you have to provide record format with constant length strings and you have to do it upfront you might be surprised. You might be also surprised your operating system understands that your data is structured and has tools supporting you with processing such data. Log retention? This is operating system feature, you can specify that log is important for 3 such files and they should rotate, opearting system will notice when you cross boundary of 5 billion records and will create next dataset (not file, there are no files, different words for many of things). If datasets (000), (001) and (002) are full, then the oldest is removed and (004) starts its own life. Oh, did I mention you can specify these rotation only once when you create dataset :)? But only if you work in backward compatible mode which is compatible with tape drives.

There are lots of topics which I haven't covered and can relate looking at other comments, I'm not sure how much of the hardware things is really true because I couldn't verify it from software side, but on the other hand I have no reason to doubt the stories other than it being so uncommon in regular computing. I hope I did not make up things as I write old memories.

One thing which is not really mainframe related is the work culture around them. It is just different, to me it seems like it is build for STEM enthusiasts, people who prefer to plan then execute. From business side there is a need to do something and see money coming, this is where is key difference, I don't know if Agile would work in world of mainframes but I doubt. Of course there are deadlines too, but overall approach is different. And because a lot of things get planned or are considered before deploying the app, the apps come with books of procedures. If error QWE123 happens then please follow these 15 steps. This is where mainframe operators come (mentioned in other comments), I don't know why this work should be done by people, terminal which accesses mainframe can be programmed around and used just like rest API (or pretty close), but there must be something extra in having some analog processing unit in the loop. Playing with emulators is not enough, you will invent your own ways of doing some things which is undesired feature when you work with mainframes, they are supposed to be low maintenance, so there is sometimes probaly a little space in mature environments to come with own custom solutions. But on the other hand they are source of many already existing concepts, e.g. blue green deployments but with option to continue started operations with old versions. Completely invisible rollout of new app version.

Mainframes are fascinating for many reasons and while I know some of the features can be done in regular environments I have bad luck in choosing jobs. Companies hiring for mainframes usually offer all required trainings, what I don't understand is why they don't offer well above the market salary for work which is far more cumbersome than other things. I received offer to work in capital city of neighbouring country of similar economic situation and the offer while was about 30-40% than my actual first job did not please me at all, and I could do regular dev job in my own capital city for a little less than that if I wanted to. I worked for one proxy dev contracting job for company with thousands of people around the world and somehow my local managers took care about opportunity to work with mainframe so it never happens, somehow it was seen as bad opportunity, I guess it was problematic for local office to deal with probable security measurements.

Re: Shortage of COBOL developers in insurance and banks

#59

Earlier quoted context omitted.

I don't blame them, some places would rather maintain their 1960s status quo than port to modern languages. It makes no sense but management is rather resistant to change.

Oh that's not the case here, management would love to get off these systems. They see the risks with the age profile of developers, staffing issues, and general ageing tech problems. Re-writing these systems in another language is almost not an option. They have been written over decades with countless undocumented projects. They are core systems that work the way the business wants (or is built around, for better or…

I lately read this wise sentence:

> “As an example, we knew that the General Ledger capability would most likely not be modernized for a very long time as changes are slow and the module is very stable,” explains Uli Homann, Corporate Vice President in the Cloud + AI Group at Microsoft. “If you were to transpose this situation to modernizing a mainframe application landscape, for applications or functionality that’s unlikely to change, it might mean that you would opt to rehost your existing COBOL, PL/I, or mainframe assembler code using capabilities from partners such as Raincode—and then eventually rewrite those applications piecemeal as COBOL or PL/I programming skills continue to get more scarce and expensive. In the meantime, though, rehosting existing code buys you time and frees-up cashflow to focus on more pressing or important modernization projects.”

https://customers.microsoft.com/en-us/story/1636062794805219...

Re: Shortage of COBOL developers in insurance and banks

#60
post #5

As many said before ; it’s not learning Cobol (or Fortran), it’s the weird IBM mainframe/OS ecosystem that is the problem. I am good at both languages, but I know very little about the ecosystem it runs in.

As a Millennial, I haven't ever seen/touched/used a mainframe. And I work in a modern Data Center. What are the pros/cons of mainframe? What makes them better than a rack housed with necessary hardware? Well, I can see cons - not so accessible as regular hardware. Impractical/impossible to have at house. So the learning curve for hobbyists/tinkerers/curious are missing first steps.

High reliability/availability, data integrity, and the insane IO are the main pros I can think of.

While there is a lot to talk about there, I think all of those points are dwarfed by the main draw: legacy support. Companies running mainframes typically have decades worth of code that is running on the mainframes and the cost/risk of migrating is enormous for questionable gains.

I definitely agree accessibility is an issue. I am pretty young myself and found COBOL to be pretty easy to learn and start developing production applications with, in many ways easier than modern systems due to lower complexity for the dev. However, the systems side of things felt very alien to me and has been a challenge to learn. I don't know how one could learn it without training at a company that has a mainframe or by taking courses with IBM.

Post reply on HN