Live data from Hacker News

Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

github.com

21–30 of 33 posts

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#21
post #17

Earlier quoted context omitted.

Do you have access to the code interpreter alpha? You can upload larger chunks of code and have it perform different tasks. One of the interesting features is that chatgpt will sometimes have some issues with its own code (python) and try to automatically correct itself.

I wish I had access to the plugins, both as a user or developer. I hate the feeling of being on the outside of the latest developments, nothing feels open about a OpenAI.

Aside from the fact you can access it at all you mean?

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#23
post #15

Earlier quoted context omitted.

> just by telling it that there's a problem how will you identify that there's a problem?

Well in a lot of cases the model is writing code that can't even be run (the red dots). Feeding it the compilation error can be done automatically and it will usually be able to at least get it running.

The last time I asked ChatGPT to write a function that I needed, it was syntactically correct right off the bat. I thought that even if it had logical errors, it would be easier to fix them one by one. It seemed like a plausible approach to take when dealing with writer's block.

However, after spending about 20 minutes fixing the details, I realized that the core logic was missing. I had only wasted precious time that I could have spent figuring it out myself.

[This comment brought to you by ChatGPT - I asked it to write a second draft of my original comment]

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#24

GPT-4 is pretty good at generating working Haskell and fixing type errors.

If I say to ChatGPT, "Write some working Haskell." it replies

"Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:"

And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell".

But that's not what working programmers do.

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#25
post #20

Earlier quoted context omitted.

> just by telling it that there's a problem how will you identify that there's a problem?

To answer your question from a practical perspective, you can try to run it, and feed back errors. See: https://news.ycombinator.com/item?id=35446171 But that's actually not what I meant. You can often just tell it "there's a problem, please fix it", or "do you see any problems" and it will be able to identify it without additional input. There no requirement that you've identified a problem, it's more of a "double c…

[deleted]

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#27
post #24

GPT-4 is pretty good at generating working Haskell and fixing type errors.

If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.

[deleted]

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#28
post #24

GPT-4 is pretty good at generating working Haskell and fixing type errors.

If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.

Here's a realistic example:

    ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format:

    {
      "items": [
 {
   "track": {
     "album": {
       "total_tracks": 9,
       "name": "playlist 1",
       "genres": [
  "Egg punk",
  "Noise rock"
       ]
     }
   }
 },
 {
   "track": {
     "album": {
       "total_tracks": 5,
       "name": "playlist 2",
       "genres": [
  "NotEgg punk",
  "NotNoise rock"
       ]
     }
   }
 }
      ]
    }
    
    Here's the code to parse the JSON and print out the album names and genres in CSV format:

    ```haskell
    {-# LANGUAGE DeriveGeneric #-}

    import Data.Aeson
    import Data.List (intercalate)
    import GHC.Generics

    data Track = Track { album :: Album }
      deriving (Show, Generic)

    data Album = Album { total_tracks :: Int, name :: String, genres :: [String] }
      deriving (Show, Generic)

    data Response = Response { items :: [Item] }
      deriving (Show, Generic)

    data Item = Item { track :: Track }
      deriving (Show, Generic)

    instance FromJSON Album
    instance FromJSON Track
    instance FromJSON Item
    instance FromJSON Response

    main :: IO ()
    main = do
      json  putStrLn "Failed to parse JSON"
 Just response -> mapM_ printAlbum $ items response

    printAlbum :: Item -> IO ()
    printAlbum item = do
      let albumName = name . album . track $ item
   albumGenres = intercalate ", " $ genres . album . track $ item
      putStrLn $ albumName ++ "," ++ albumGenres
    ```

    Assuming the JSON is stored in the file "example.json", running this program will output:

    ```
    playlist 1,Egg punk, Noise rock
    playlist 2,NotEgg punk, NotNoise rock
    ```

    This is the album name and genres from each item in the "items" list of the JSON, separated by a comma.

    ChatGPT> gpt1.hs:27:23: error:
 • Couldn't match type ‘[Char]’
    with ‘Data.ByteString.Lazy.Internal.ByteString’
   Expected: Data.ByteString.Lazy.Internal.ByteString
     Actual: String
 • In the first argument of ‘decode’, namely ‘json’
   In the expression: decode json :: Maybe Response
   In an equation for ‘parsed’: parsed = decode json :: Maybe Response
       |
    27 |   let parsed = decode json :: Maybe Response
       |
    
    The error message is indicating that there's a type mismatch between the `json` string and the expected `ByteString` type needed by the `decode` function.

    To fix this, you can use the `Data.ByteString.Lazy.Char8` module to create a `ByteString` from the `json` string:

    ```haskell
    import qualified Data.ByteString.Lazy.Char8 as BS

    -- ...
    main :: IO ()
    main = do
      json 
And it works fine:

    $ nix-shell -p "haskellPackages.ghcWithPackages (ps: with ps; [aeson])" --run "runhaskell gpt1.hs"
    playlist 1,Egg punk, Noise rock
    playlist 2,NotEgg punk, NotNoise rock

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#29
post #24

Earlier quoted context omitted.

If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.

Here's a realistic example: ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format: { "items": [ { "track": { "album": { "total_tracks": 9, "name": "playlist 1", "genres": [ "Egg punk", "Noise rock" ] } } }, { "track": { "album": { "total_tracks": 5, "name": "playlist 2", "genres": [ "NotEgg punk", "NotNoise rock" ] } } } ] } Here's the code to parse…

I don't understand what use this is.

I mean, let's say you really do have to parse this format for something at work.

So you run it on a GB of similar data (meaning not literally specifically a GB, but well more than you can reasonably verify by hand).

Then what? How do you know it works fine, or if it obviously crashes, what to do next?

Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities

#30
post #24

Earlier quoted context omitted.

If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.

Here's a realistic example: ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format: { "items": [ { "track": { "album": { "total_tracks": 9, "name": "playlist 1", "genres": [ "Egg punk", "Noise rock" ] } } }, { "track": { "album": { "total_tracks": 5, "name": "playlist 2", "genres": [ "NotEgg punk", "NotNoise rock" ] } } } ] } Here's the code to parse…

[deleted]
Post reply on HN