Live data from Hacker News

AI migrated legacy COBOL programs to Java, bugs included

arxiv.org

91–100 of 108 posts

Re: AI migrated legacy COBOL programs to Java, bugs included

#91
I just asked Google AI how LLMs translate COBOL code containing GOTOs into Java, a language that doesn't have GOTO. It gave several examples, one of them being this:

COBOL code:

  PROCESS-DATA.
      ADD 1 TO COUNTER.
      DISPLAY COUNTER.
      IF COUNTER 
Java translation:

  while (counter 
If "counter" is 10 on entry, the COBOL code prints 11 while the Java code prints nothing. So not only keeping old bugs, but apparently introducing new ones too!

I wrote COBOL code for a few years at a job when I was a teenager. What makes legacy COBOL code difficult IMO is it can sometimes be very hard to maintain a mental execution state when examining the code, for several reasons:

1. all variables are global, aka, WORKING-STORAGE. You list all the variables used in the program and they are accessible to the entire program.

2. programs are divided into paragraphs. Control normally flows sequentially top to bottom through paragraphs, one executing after another. Except that the PERFORM statement can drastically alter this normal flow control, and you can't tell by looking at a paragraph how it will be executed. To do that, you have to look at all PERFORM statements that mention this paragraph or any paragraph physically before it, because in COBOL you can say PERFORM PARA1 THROUGH PARA27. If PARA13 is physically between PARA1 and PARA27, it's potentially going to get executed.

3. In true legacy COBOL, before structured COBOL was a thing (circa 1985), the main control flow statement in addition to PERFORM was GOTO. Lots of flag setting, and lots of GOTOs. So in the previous example, you can't tell if PARA13 is going to get executed because any prior statement might be a GOTO PARA14, skipping execution of PARA13. But even worse, you are still under the influence of the PERFORM THRU, so after PARA27 is executed, control returns to the statement following the PERFORM THRU, wherever that was. But if you GOTO PARA27, without being under a PERFORM THRU, then PARA27 is executed followed by the next sequential paragraph. Trying to figure this out statically by looking at the program can be very difficult, especially considering PERFORMs that are nested at runtime but may not be anywhere near each other in a code listing.

Re: AI migrated legacy COBOL programs to Java, bugs included

#92
post #23

Earlier quoted context omitted.

> AI is not deterministic, it will be making tons of mistakes. From the paper: > The COBOL source is passed through an internal deterministic Migrator to produce a generated Java target. Also, humans are not deterministic either. Give the same COBOL -> Java translation to multiple developers and each will come up with a different solution. Heck, even the same developer will produce a different output for the same tas…

So they solved the "make no mistakes" problem? They deserve a nobel prize.

Deterministic != Correct

Re: AI migrated legacy COBOL programs to Java, bugs included

#93
post #65

Earlier quoted context omitted.

> Now, no one does. I got a chuckle over this, but in my experience working on legacy systems, this is the case already. Using AI to translate/explain the code brings extra understanding.

Can it really explain the code if the information is not there anymore ? It was in a binder that got shredded 40 years ago by accident. More like it will hallucinate some explanation & cause even more confusion.

The code is the information. If you need binders of info then it wasn’t codified

Re: AI migrated legacy COBOL programs to Java, bugs included

#94

Before, only the senior cobol programmers at the company understood and knew the codebase. Now, no one does. I can see the allure of moving away from a legacy cobol system. But an AI rewrite doesn't actually solve any of the issues with having a legacy cobol codebase. You just have a new system no one knows or understands.

It seems to me that some companies are looking to move off COBOL and mainframes to say Linux and Java, because that is what someone was employed to do, not because it's necessarily a good idea.

I think COBOL systems would be plenty maintainable if the owners of the systems invested in maintaining them. If companies just let people retire, attempt to hire replacements with unremarkable salaries, then maintenance is a problem.

If they paid salaries that reflected how critical this work is, I don't think it would be that big a problem to find and keep good developers.

We have critical economic infrastructure being treated as annoyance and a cost centre.

Re: AI migrated legacy COBOL programs to Java, bugs included

#95

Before, only the senior cobol programmers at the company understood and knew the codebase. Now, no one does. I can see the allure of moving away from a legacy cobol system. But an AI rewrite doesn't actually solve any of the issues with having a legacy cobol codebase. You just have a new system no one knows or understands.

Java should have significantly better tools for trying to understand a program than cobol.

Re: AI migrated legacy COBOL programs to Java, bugs included

#96
post #94

Before, only the senior cobol programmers at the company understood and knew the codebase. Now, no one does. I can see the allure of moving away from a legacy cobol system. But an AI rewrite doesn't actually solve any of the issues with having a legacy cobol codebase. You just have a new system no one knows or understands.

It seems to me that some companies are looking to move off COBOL and mainframes to say Linux and Java, because that is what someone was employed to do, not because it's necessarily a good idea. I think COBOL systems would be plenty maintainable if the owners of the systems invested in maintaining them. If companies just let people retire, attempt to hire replacements with unremarkable salaries, then maintenance is a…

I firmly believe that any thoughtful rewrite is good. Most companies just don't have resources for that. At least, it serves as a security review, but also refreshes team knowledge of the system. They usually also improve documentation if it was wrong.

Re: AI migrated legacy COBOL programs to Java, bugs included

#97
post #89

Earlier quoted context omitted.

> And we can also implement llm inference deterministically if we want, it’s just that it’s not worth the loss in performance to do it. My understanding is thats not possible (different from being practical), wonder if you have any literature, research to back up that claim?

This one from thinking machines: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... Was discussed a lot also here on hn

Thanks for sharing, it was really good read. The authors in the article have done a good job for sure to validate their hypothesis and make it work for a particular subset of problem.

1. Local hardware, no networking or HTTP requests 2. No other parallel requests 3. 1000 runs bounded by 1000 tokens

I have worked a little bit in academia and I am not a big fan of the way favorable samples for the hypothesis are kept and unfavorable ones are thrown away. I might be wrong here, however its highly unlikely that the team would have just worked with one query. Chances are that a lot of different prompts with varying number of runs would have been tried to see what works and supports the hypothesis.

Re: AI migrated legacy COBOL programs to Java, bugs included

#98
post #69

Earlier quoted context omitted.

AI doesn't need to be deterministic.

But, hear me out, a COBOL to Java migration absolutely needs to be deterministic. Surviving COBOL programs run a lot of sensitive operations. As a side note, I can't fathom why they chose Java. It's easier to teach COBOL to a competent engineer than to rewrite everything in a language that encourages onion architectures. Why not Go, for example?

Java as a programming language doesn't enforce any architecture. There is no difference to go. Its a system of folders.

The tests need to be green, the math needs to math, but the code can look a little bit different here or there.

I don't think its easy at all to educate someone on COBOL. Its carrier suicide if you want to ever do anything else again after that gig. I was writing java for years, switched for 2 years to php for another company and had a few companies complaining about these 2 years...

Re: AI migrated legacy COBOL programs to Java, bugs included

#99

Before, only the senior cobol programmers at the company understood and knew the codebase. Now, no one does. I can see the allure of moving away from a legacy cobol system. But an AI rewrite doesn't actually solve any of the issues with having a legacy cobol codebase. You just have a new system no one knows or understands.

Java should have significantly better tools for trying to understand a program than cobol.

It has had for plenty of years now, however many think vi and emacs is enough.

Re: AI migrated legacy COBOL programs to Java, bugs included

#100

Earlier quoted context omitted.

Wasn't COBOL invented to read like business prose and be easily understood by programmers and non-programmers alike? If developers are able to write COBOL code no one understands than it failed its main objective. IMHO every large software system should be developed on the basis of clear and concise functional specifications so everyone can understand what the software does and how it does so at a higher functional l…

Cobol code isn't esoteric, but it is designed in a way that is unambiguous and clear about its control flow, but very verbose and tedious to read/write. Translating from Cobol to a modern programming language is trivial for LLMs for that reason because the lack of any hidden state functionalities like generics for example.

Modern Cobol has most of the goodies, it is much more feature rich than Go.
Post reply on HN