Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How well can LLMs write COBOL?

bloop.ai

Post reply on HN
Print view View original

31–40 of 63 posts

Re: How well can LLMs write COBOL?

#31

Post by bongodongobob »

allusernamesare wrote: post #7

On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.

Why do people always add this "blah blah hallucinations and critical systems"? 1. People write bad and buggy code. 2. You act like we're just blindly throwing untested code at production systems from LLMs.

It's just intellectually dishonest to talk this way.

They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.

Re: How well can LLMs write COBOL?

#32

Post by kamma4434 »

sanxiyn wrote: post #20
mrbombastic wrote: post #17

Earlier quoted context omitted.

I wonder if a better goal would be valid translation of the COBOL into X language. Obviously that has its own can of worms but it seems like our goal generally should be getting critical systems to more modern languages that can be maintained more easily.

In 2009, 4 million lines of COBOL were migrated to Java using an automatic translator. https://www.infoq.com/news/2009/07/cobol-to-java/

The problem is not translation - once it is ported to Java, who will maintain it? It has no specs, no tests, and it’s a spaghetti mess. Is any weirdness a bug or a feature? Nobody knows. Least of all the body-rental remote devs you hired to maintain it.

Re: How well can LLMs write COBOL?

#33

Post by CuriouslyC »

ripvanwinkle wrote: post #6

It would be interesting to feed it a formal language specification of some language it hasn't seen and then ask it write code and see how it does. That could be a test of reasoning and reading comprehension

I've been thinking about a benchmark designed this way for a while. It doesn't even need to be code, particularly, it could be basic reasoning problems. The key is that you define a new, random language that has never before been seen (maybe it has statistical similarity to existing languages, maybe not), create a translation key, then ask a question in that language.

Re: How well can LLMs write COBOL?

#34

Post by bigEnotation »

treebeard901 wrote: post #26

One thing that makes OpenAI so valuable over time is how they can take all of this expert input from the rush to test out the new technology and use that to exponentially improve the next model. You have to think, just like we see here, experts in their field who know all of these edge cases or other fundamental aspects of what makes their own companies or projects valuable and they tend to hand that over in the proc…

I think you’re forgetting about the use case where the LLM returns something partially correct to a discerning expert, who is still able to use the response, but does not bother with a message like “btw I had to do X to make your suggestions usable”.

Re: How well can LLMs write COBOL?

#35

Post by CuriouslyC »

ape4 wrote: post #19
ripvanwinkle wrote: post #6

It would be interesting to feed it a formal language specification of some language it hasn't seen and then ask it write code and see how it does. That could be a test of reasoning and reading comprehension

Reasoning vs being a completion engine (I could make a guess at how well that would work)

Reasoning is a form of completion (logical), the problem is that LLMs aren't language agnostic in their learned semantic reasoning.

Re: How well can LLMs write COBOL?

#36

Post by gnatolf »

andy99 wrote: post #25

I've asked chatGPT a fair number of Fortran questions. There are differences - Fortran is still in use lots of places, there are forums and documentation sites (though presumably that's true for COBOL). But compare to python, there is way less info out there on how to do different things, for example little Stack Overflow content. I'd say I has mixed results, definitely chatGPT knows the language and can give example…

Fortran also has the disadvantage of numerous separate dialects/flavours that plenty of times can't be mixed. And rarely is the exact style mentioned in random code found somewhere.

It got a lot better with Fortran 95 and newer, but in the old world of e.g. lahey compilers and custom commands only available there, any LLM has failed me consistently to stick to these intricacies. I can't even blame them, when asking humans questions about these topics, you'll get all sorts of answers that are equally close to a correct solution, but almost never precisely correct.

Re: How well can LLMs write COBOL?

#37

Post by SonOfLilit »

skissane wrote: post #30
danenania wrote: post #21

This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases. I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-…

> Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that) Unless GPT-4's training data i…

I've been using GPT4 to help me navigate a mainframe and a COBOL codebase and it knows far more than what my googling abilities manage to fish up in forums. It's actually surprisingly good at surprisingly deep mainframe topics.

Re: How well can LLMs write COBOL?

#38

Post by fragmede »

bongodongobob wrote: post #31
allusernamesare wrote: post #7

On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.

Why do people always add this "blah blah hallucinations and critical systems"? 1. People write bad and buggy code. 2. You act like we're just blindly throwing untested code at production systems from LLMs. It's just intellectually dishonest to talk this way. They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.

Look, we can't all just be realistic about a thing that's going to take our jobs, so we have to lean on tired old excuses instead. Instead of being reasonable, why don't you pick a team - for or against, and then fight about it on Internet forums because, well, what else are you going to do while the build compiles? Look at cat videos?

Re: How well can LLMs write COBOL?

#39

Post by bongodongobob »

fragmede wrote: post #38
bongodongobob wrote: post #31

Earlier quoted context omitted.

Why do people always add this "blah blah hallucinations and critical systems"? 1. People write bad and buggy code. 2. You act like we're just blindly throwing untested code at production systems from LLMs. It's just intellectually dishonest to talk this way. They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.

Look, we can't all just be realistic about a thing that's going to take our jobs, so we have to lean on tired old excuses instead. Instead of being reasonable, why don't you pick a team - for or against, and then fight about it on Internet forums because, well, what else are you going to do while the build compiles? Look at cat videos?

I generate my own custom cat videos with AI now tyvm.

Re: How well can LLMs write COBOL?

#40

Post by giantrobot »

bongodongobob wrote: post #31
allusernamesare wrote: post #7

On one hand, it'd be super cool to solve the talent shortage in the field, also Cobol code isn't very pleasant to write. On the other, I'm not sure I'd want tools known for poor code quality, and hallucinations, to write these super critical systems. Guess there might be a copilotesque productivity booster for human developers, but I think these systems are some of the last places I'd want LLMs to contribute.

Why do people always add this "blah blah hallucinations and critical systems"? 1. People write bad and buggy code. 2. You act like we're just blindly throwing untested code at production systems from LLMs. It's just intellectually dishonest to talk this way. They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.

> It's just intellectually dishonest to talk this way.

> They will still be helpful but we obviously need to test before we add code into systems. It goes without saying.

It's not intellectually dishonest at all. It's an issue of conditioning. There's a class of developers that blindly copy and paste code from StackOverflow or the first hit on Google. They're the same class that will uncritically copy and paste ChatGPT answers.

ChatGPT is worse than SO because it's adaptive. If someone pastes in a SO answer and it doesn't immediately work the developer has to at least engage with the code. ChatGPT can be asked to refine its hallucination until it parses/compiles.

The class of developer blindly copying and pasting answers will not have the expertise to spot hallucinations or likely even fix the inevitable bugs they introduce. Additionally ChatGPT by its nature elides the source of its answers. At the very least a SO answer has some provenance. Not only the poster but some social signally through votes that the answer is legitimate.

ChatGPT answers don't have any of that. It will also happily hallucinate references.

Conditioning junior developers and learners to rely on and trust AI coding is setting them up to fail. It's also going to stunt their growth as developers because they'll never gain any domain knowledge. In the meantime they'll be unknowingly sabotaging products with legit looking but broken code.

Post reply on HN