Considering MTOB(Machine Translation from One Book) result, where LLM in-context learns a new language from a grammar book, I wonder how LLMs fare given, say, GnuCOBOL Programmer's Guide PDF, which is easily available. It would be an interesting addition to the benchmark.
How well can LLMs write COBOL?
11–20 of 63 posts
Re: How well can LLMs write COBOL?
#12I suspect there were relatively few training examples for COBOL. It would be interesting to see the results for a system which had a significant number of such examples in the training set.
There is probably a business opportunity for an AI company to build private LLMs for large enterprises trained on their own COBOL code bases. They won't find much available as open source, and individual companies tend to have significantly different coding styles tied to COBOL versions and database schemas.
Yeah I think Facebook and Google are already doing that internally.
Re: How well can LLMs write COBOL?
#13I tried to get ChatGPT to write LLVM IR last year. The results were interesting: the LLM wrote superficially correct-looking IR, but it ultimately failed to grasp the concept of SSA, as it kept trying to write to SSA registers. ChatGPT can generalize across language syntax reasonably well, but it doesn't understand deeper differences in language semantics.
Re: How well can LLMs write COBOL?
#14On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.
That would remove some portion of the human coders, too.
Re: How well can LLMs write COBOL?
#15I tried to get chatgpt to write 6502 assembler for the 1980s 8-bit home computer the BBC Micro. It was game, but clueless
I tried to get it to write 6502 assembler for the NES. As I recall it mistakenly thought that there was a BIOS containing a print function.
Re: How well can LLMs write COBOL?
#16Since the LLM sometimes generates invalid COBOL a simple practical solution would be to use an API and allow it to test its code with GnuCOBOL, feed the output and have it try again a couple of times. I wonder what the updated benchmarks would be in that setting.
Re: How well can LLMs write COBOL?
#17On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.
I wonder if a better goal would be valid translation of the COBOL into X language. Obviously that has its own can of worms but it seems like our goal generally should be getting critical systems to more modern languages that can be maintained more easily.
Re: How well can LLMs write COBOL?
#18I suspect there were relatively few training examples for COBOL. It would be interesting to see the results for a system which had a significant number of such examples in the training set.
Indeed, even the author wasn't aware of modern COBOL, with IDE tooling and OOP capabilities, focusing on the classical micros instead.
Re: How well can LLMs write COBOL?
#19It would be interesting to feed it a formal language specification of some language it hasn't seen and then ask it write code and see how it does. That could be a test of reasoning and reading comprehension
Reasoning vs being a completion engine (I could make a guess at how well that would work)
Re: How well can LLMs write COBOL?
#20On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.
I wonder if a better goal would be valid translation of the COBOL into X language. Obviously that has its own can of worms but it seems like our goal generally should be getting critical systems to more modern languages that can be maintained more easily.
In 2009, 4 million lines of COBOL were migrated to Java using an automatic translator.