Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How well can LLMs write COBOL?

bloop.ai

Post reply on HN
Print view View original

21–30 of 63 posts

Re: How well can LLMs write COBOL?

#21

Post by danenania »

This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex

It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases.

I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains. In any case, I'll try out this benchmark and see how it goes.

Re: How well can LLMs write COBOL?

#23

Post by wvenable »

pcwalton wrote: post #13

I tried to get ChatGPT to write LLVM IR last year. The results were interesting: the LLM wrote superficially correct-looking IR, but it ultimately failed to grasp the concept of SSA, as it kept trying to write to SSA registers. ChatGPT can generalize across language syntax reasonably well, but it doesn't understand deeper differences in language semantics.

I tried to get ChatGPT to write 6502 assembly and it had similar issues.

Re: How well can LLMs write COBOL?

#24

Post by saurik »

wvenable wrote: post #23
pcwalton wrote: post #13

I tried to get ChatGPT to write LLVM IR last year. The results were interesting: the LLM wrote superficially correct-looking IR, but it ultimately failed to grasp the concept of SSA, as it kept trying to write to SSA registers. ChatGPT can generalize across language syntax reasonably well, but it doesn't understand deeper differences in language semantics.

I tried to get ChatGPT to write 6502 assembly and it had similar issues.

FWIW, I had similar issues trying to get both it and Claude to help me with x86 assembly: it kept thinking if it added another * or some more parentheses it could get some impossible assembly to work.

Re: How well can LLMs write COBOL?

#25

Post by andy99 »

I've asked chatGPT a fair number of Fortran questions. There are differences - Fortran is still in use lots of places, there are forums and documentation sites (though presumably that's true for COBOL). But compare to python, there is way less info out there on how to do different things, for example little Stack Overflow content.

I'd say I has mixed results, definitely chatGPT knows the language and can give examples but I've also had a lot of frustrating things it wasn't able to resolve.

Re: How well can LLMs write COBOL?

#26

Post by treebeard901 »

One thing that makes OpenAI so valuable over time is how they can take all of this expert input from the rush to test out the new technology and use that to exponentially improve the next model. You have to think, just like we see here, experts in their field who know all of these edge cases or other fundamental aspects of what makes their own companies or projects valuable and they tend to hand that over in the process.

It's kind if interesting how far this could be taken with all kinds of valuable information from people and companies, especially code and business logic. Everyone wants to say AI this and AI that to keep up with the times and they all keep dumping all of this valuable data in for free.

Then consider the authentication method used tied to this data collection and if you are using a company emial, etc, it helps them weed out the garbage too.

I guess we can't fight progress...

Re: How well can LLMs write COBOL?

#27

Post by arittr »

danenania wrote: post #21

This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases. I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-…

This looks great! Can’t wait to try it out today

Re: How well can LLMs write COBOL?

#28

Post by CuriouslyC »

treebeard901 wrote: post #26

One thing that makes OpenAI so valuable over time is how they can take all of this expert input from the rush to test out the new technology and use that to exponentially improve the next model. You have to think, just like we see here, experts in their field who know all of these edge cases or other fundamental aspects of what makes their own companies or projects valuable and they tend to hand that over in the proc…

That might be true, but my feeling so far is that OpenAI doesn't want to do what they'd need to do to make any one product actually good, so they're going to keep bouncing from AI thing to thing, making foundation models that have a lot of wow factor but can't really deliver on their promise because they're too closed off to integrate into a workflow that might alleviate the issues.

Other people will try to build on OpenAI stuff, find that it's not quite good enough and OpenAI doesn't care to really make it good enough because it's a lot of work, and it won't be until we get competitors that take less sexy model tech and take the time to make it REALLY GOOD at certain things that AI really makes good on its promise. I'm guessing that will be driven by people taking open source tools that are ~80% of the way there and really building a system and domain logic around it to make it excellent.

Re: How well can LLMs write COBOL?

#29

Post by bongodongobob »

saurik wrote: post #24
wvenable wrote: post #23

Earlier quoted context omitted.

I tried to get ChatGPT to write 6502 assembly and it had similar issues.

FWIW, I had similar issues trying to get both it and Claude to help me with x86 assembly: it kept thinking if it added another * or some more parentheses it could get some impossible assembly to work.

Reminds me of adding *'s and &'s until my C++ code worked in college.

Re: How well can LLMs write COBOL?

#30

Post by skissane »

danenania wrote: post #21

This looks interesting. I'm working on an OpenAI-based tool for coding tasks that are too complex for ChatGPT - https://github.com/plandex-ai/plandex It's working quite well for me, but it definitely needs some time spent on benchmarking and ironing out edge cases. I'm especially curious how it will do on more "obscure" languages. Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-…

> Not that Cobol is obscure exactly--I suppose there's probably quite a bit of it in GPT-4's training considering how pervasive it is in some domains

There is a huge amount of COBOL code in existence – but, almost all of it is non-public code used to run business and governments. Very little of it is publicly source-available (whether open source or something more restrictive than that)

Unless GPT-4's training data includes non-public code bases (I doubt it), it likely has rather little COBOL code in it

Post reply on HN