I tried to get chatgpt to write 6502 assembler for the 1980s 8-bit home computer the BBC Micro. It was game, but clueless
It can write 6502, but you have to know 6502 yourself to coach it.
How well can LLMs write COBOL?
51–60 of 63 posts
Re: How well can LLMs write COBOL?
#52Earlier quoted context omitted.
> Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that) Unless GPT-4's training data i…
But it probably read all the books ever published on COBOL.
Almost certainly not. There are heaps of books in libraries which nobody has scanned yet, including many on COBOL. No LLM has read those.
Whether that makes a difference depends. If you are dealing with mainstream modern COBOL (like recent versions of MicroFocus or IBM Enterprise), it probably won't. If you are dealing with some obscure legacy COBOL dialect, odds are high there is some very helpful printed book which nobody has scanned.
Re: How well can LLMs write COBOL?
#53Earlier quoted context omitted.
I've been using GPT4 to help me navigate a mainframe and a COBOL codebase and it knows far more than what my googling abilities manage to fish up in forums. It's actually surprisingly good at surprisingly deep mainframe topics.
No doubt its training data contains a lot of IBM manuals, probably even some commercial books on relevant topics, maybe even the contents of some of the forums you mention – and all that could be enough to correctly answer your questions. However, for languages like Python, Java, C, C++, JavaScript, Go, etc, it also contains untold millions of lines of code slurped from places like GitHub. Whereas, I really doubt it…
Look, nobody is going to perform as well on complicated programming talks using cobol as with python. But knowing everything you said, I was amazed at how good it was. Try it.
Also, nobody needs to do complicated coffee talks with cobol, it wasn't meant for it. What we do need a lot of is translating cobol to python or Java.
Re: How well can LLMs write COBOL?
#54Earlier quoted context omitted.
No doubt its training data contains a lot of IBM manuals, probably even some commercial books on relevant topics, maybe even the contents of some of the forums you mention – and all that could be enough to correctly answer your questions. However, for languages like Python, Java, C, C++, JavaScript, Go, etc, it also contains untold millions of lines of code slurped from places like GitHub. Whereas, I really doubt it…
Look, nobody is going to perform as well on complicated programming talks using cobol as with python. But knowing everything you said, I was amazed at how good it was. Try it. Also, nobody needs to do complicated coffee talks with cobol, it wasn't meant for it. What we do need a lot of is translating cobol to python or Java.
To rephrase my point: the gap between the best an LLM can do and the best an experienced human can do, is likely larger for COBOL than for more mainstream languages, simply because LLMs have a lot more opportunities to gain "experience" with those more mainstream languages than they do with COBOL.
What you are saying may well all be true, but it doesn't contradict what I'm saying.
Re: How well can LLMs write COBOL?
#55Earlier quoted context omitted.
I wonder if a better goal would be valid translation of the COBOL into X language. Obviously that has its own can of worms but it seems like our goal generally should be getting critical systems to more modern languages that can be maintained more easily.
In 2009, 4 million lines of COBOL were migrated to Java using an automatic translator. https://www.infoq.com/news/2009/07/cobol-to-java/
Thanks for sharing! Will need to dig into this
Re: How well can LLMs write COBOL?
#56Earlier quoted context omitted.
No doubt its training data contains a lot of IBM manuals, probably even some commercial books on relevant topics, maybe even the contents of some of the forums you mention – and all that could be enough to correctly answer your questions. However, for languages like Python, Java, C, C++, JavaScript, Go, etc, it also contains untold millions of lines of code slurped from places like GitHub. Whereas, I really doubt it…
Look, nobody is going to perform as well on complicated programming talks using cobol as with python. But knowing everything you said, I was amazed at how good it was. Try it. Also, nobody needs to do complicated coffee talks with cobol, it wasn't meant for it. What we do need a lot of is translating cobol to python or Java.
Are you hiring for it? I don't know COBOL, but do know python and Java to some extent, and enjoy esoteric legacy problems
Re: How well can LLMs write COBOL?
#57Earlier quoted context omitted.
Look, nobody is going to perform as well on complicated programming talks using cobol as with python. But knowing everything you said, I was amazed at how good it was. Try it. Also, nobody needs to do complicated coffee talks with cobol, it wasn't meant for it. What we do need a lot of is translating cobol to python or Java.
Are you hiring for it? I don't know COBOL, but do know python and Java to some extent, and enjoy esoteric legacy problems
You can know learn Cobol online, see IBM's offering in coursera, e.g.: https://www.coursera.org/learn/cobol-programming-vscode
Re: How well can LLMs write COBOL?
#58This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases. I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-…
> I'm especially curious how it will do on more "obscure" languages.
There’s definitely a lack of training data and ability (but unfortunately not confidence) in less widespread languages. It’s quite bad at pinescript, confusing versions of the language and producing unrunnable code, and being unable to correct it when given feedback
Re: How well can LLMs write COBOL?
#59I tried to get ChatGPT to write LLVM IR last year. The results were interesting: the LLM wrote superficially correct-looking IR, but it ultimately failed to grasp the concept of SSA, as it kept trying to write to SSA registers. ChatGPT can generalize across language syntax reasonably well, but it doesn't understand deeper differences in language semantics.
I tried to get ChatGPT to write 6502 assembly and it had similar issues.
I got this from Gemini:
> Gemini:
> JSON: In JSON, double quotes are used to delimit strings. If a double quote needs to be included within the string itself, it's escaped by doubling it (e.g., `"This is a string with a "quote" in it."`).
(And, just an editorial note: there are no backslashes in Gemin's response / HN is displaying it correctly. Gemini's response is nonsense — wrong, but also not even internally consistent.)
> Me:
> Does your JSON example parse as valid JSON?
> Gemini:
> Yes, the JSON example I provided ("This is a string with a "quote" in it.") is actually valid JSON.
> The reason it's valid is because the doubled quote within the string is interpreted as an escaped character, representing a single literal quote within the string itself. The JSON parser recognizes the double quotes at the beginning and end as delimiters for the entire string, not the escaped quote within.
(Its example, is, of course, quite malformed.)
Re: How well can LLMs write COBOL?
#60Earlier quoted context omitted.
The problem is not translation - once it is ported to Java, who will maintain it? It has no specs, no tests, and it’s a spaghetti mess. Is any weirdness a bug or a feature? Nobody knows. Least of all the body-rental remote devs you hired to maintain it.
which is worse than the same situation, but in COBOL, how exactly?
Because you have the same mess plus all the mess and hard to debug incompatibilities and weirdness, added by the translator.
What people don’t get about cobol is that it is a language you can learn in a few days. It is meant to be readable for non-technical users so Cobol itself is not the issue. Once you get past the meaning of sections, and the implicit defaults that are different from modern programming languages, it is way easier than, say, C.