A while back, I asked a question here, roughly, why hasn’t someone written, say, a C to COBOL translator? Such a program might take a lot of work, but it seemed to me that with an impending dearth of COBOL programmers, there would be demand for such an app. I was informed that there were so many different COBOLs in use that the output of such a program would STILL have to be tended to by an experienced programmer in the output dialect desired. This is just the Copilot situation.
How well can LLMs write COBOL?
41–50 of 63 posts
Re: How well can LLMs write COBOL?
#42I suspect there were relatively few training examples for COBOL. It would be interesting to see the results for a system which had a significant number of such examples in the training set.
Indeed, even the author wasn't aware of modern COBOL, with IDE tooling and OOP capabilities, focusing on the classical micros instead.
The issue is less COBOL the language and more the literal business logic the COBOL is encoding. You can learn the COBOL language as easily as any other. What you can't learn as easily are the accounting rules, administration policies, and regulations any major COBOL codebase is implementing.
You'll be able to see the code multiplying a dollar value by 0.03 but not necessarily know that this is because of some statutory requirement of some Minnesota tax code that only applies to industries producing both left and right handed monkey wrenches but only if the company was incorporated before 1975. That obscure law isn't referenced in any documentation but was found by an accountant in 1982. The change was made to the code but only referenced in a paper memo with a small distribution list but all of those memos were shredded after being archived for ten years.
ChatGPT can't really help document code that's decades old and doesn't have any references to the why of the code. The how is straightforward but rarely as important as the why.
Re: How well can LLMs write COBOL?
#43A while back, I asked a question here, roughly, why hasn’t someone written, say, a C to COBOL translator? Such a program might take a lot of work, but it seemed to me that with an impending dearth of COBOL programmers, there would be demand for such an app. I was informed that there were so many different COBOLs in use that the output of such a program would STILL have to be tended to by an experienced programmer in…
As much as everyone likes to poke fun at COBOL, the language itself really isn't the problem with maintaining/updating old COBOL systems. It's old, but it's not that bad.
The real problem is the entire ecosystem around those systems. Remember, a lot of COBOL software dates back to a time before things like relational databases. You'll be working with flat files that might, if you're very lucky, have column and record separators and useful names/documentation explaining what they are. If you're unlucky you'll have to figure out field widths from the code and infer what the fields are based on their actual usage. Oh and if you get it wrong you just messed up something related to payroll or financial compliance; enjoy the punishing fines.
That kind of stuff, more than the language, is the reason nobody wants to touch old COBOL systems.
Re: How well can LLMs write COBOL?
#44Earlier quoted context omitted.
In 2009, 4 million lines of COBOL were migrated to Java using an automatic translator. https://www.infoq.com/news/2009/07/cobol-to-java/
The problem is not translation - once it is ported to Java, who will maintain it? It has no specs, no tests, and it’s a spaghetti mess. Is any weirdness a bug or a feature? Nobody knows. Least of all the body-rental remote devs you hired to maintain it.
which is worse than the same situation, but in COBOL, how exactly?
Re: How well can LLMs write COBOL?
#45wow! thats a memory lane i hope never revisit!
Re: How well can LLMs write COBOL?
#46Since the LLM sometimes generates invalid COBOL a simple practical solution would be to use an API and allow it to test its code with GnuCOBOL, feed the output and have it try again a couple of times. I wonder what the updated benchmarks would be in that setting.
The general approach seems to work anyway. I tried it out with ChatGPT 3.5 and an online Cobol compiler[0], manually feeding back the output, and it managed to produce a working program on the 10th attempt (that displays the first 10 Fibonacci numbers).
Edit: Well, maybe. With the example from the article it wasn't as successful.
Re: How well can LLMs write COBOL?
#47Earlier quoted context omitted.
Indeed, even the author wasn't aware of modern COBOL, with IDE tooling and OOP capabilities, focusing on the classical micros instead.
The issue is less COBOL the language and more the literal business logic the COBOL is encoding. You can learn the COBOL language as easily as any other. What you can't learn as easily are the accounting rules, administration policies, and regulations any major COBOL codebase is implementing. You'll be able to see the code multiplying a dollar value by 0.03 but not necessarily know that this is because of some statuto…
Not knowing a statutory law expressed in code is not just a COBOL thing.
Re: How well can LLMs write COBOL?
#48This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases. I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-…
> Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that) Unless GPT-4's training data i…
But it probably read all the books ever published on COBOL.
Re: How well can LLMs write COBOL?
#49Earlier quoted context omitted.
Why do people always add this "blah blah hallucinations and critical systems"? 1. People write bad and buggy code. 2. You act like we're just blindly throwing untested code at production systems from LLMs. It's just intellectually dishonest to talk this way. They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.
> It's just intellectually dishonest to talk this way. > They will still be helpful but we obviously need to test before we add code into systems. It goes without saying. It's not intellectually dishonest at all. It's an issue of conditioning. There's a class of developers that blindly copy and paste code from StackOverflow or the first hit on Google. They're the same class that will uncritically copy and paste ChatG…
I should be worried that the very worst developers might paste bad code from ChatGPT and that's why it's dangerous? Looks an awful lot like mental gymnastics to me.
Re: How well can LLMs write COBOL?
#50Earlier quoted context omitted.
> Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that) Unless GPT-4's training data i…
I've been using GPT4 to help me navigate a mainframe and a COBOL codebase and it knows far more than what my googling abilities manage to fish up in forums. It's actually surprisingly good at surprisingly deep mainframe topics.
No doubt its training data contains a lot of IBM manuals, probably even some commercial books on relevant topics, maybe even the contents of some of the forums you mention – and all that could be enough to correctly answer your questions.
However, for languages like Python, Java, C, C++, JavaScript, Go, etc, it also contains untold millions of lines of code slurped from places like GitHub. Whereas, I really doubt it contains anywhere remotely near as much COBOL code, just because you look for COBOL code on GitHub public repos, you will find very little – the vast majority of COBOL code is in-house or vendor business software, and few seem to want to make that stuff public – and what COBOL code GitHub has is mostly toy exercises or ancient stuff, not examples of significant contemporary production code. The only way OpenAI is going to get a substantial quantity of that is if multiple private parties (such as banks) give them access to their COBOL code bases – not impossible, but absent some public info saying it has happened, it seems more likely it hasn't.
I expect GPT-4 (or any LLM) is not going to perform as well on complicated programming tasks for COBOL compared to other languages. For more mainstream languages, it has millions of examples to help it do a better job, for COBOL it likely doesn't.