Earlier quoted context omitted.
Do you have access to the code interpreter alpha? You can upload larger chunks of code and have it perform different tasks. One of the interesting features is that chatgpt will sometimes have some issues with its own code (python) and try to automatically correct itself.
I wish I had access to the plugins, both as a user or developer. I hate the feeling of being on the outside of the latest developments, nothing feels open about a OpenAI.
Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
21–30 of 33 posts
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#22Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#23Earlier quoted context omitted.
> just by telling it that there's a problem how will you identify that there's a problem?
Well in a lot of cases the model is writing code that can't even be run (the red dots). Feeding it the compilation error can be done automatically and it will usually be able to at least get it running.
However, after spending about 20 minutes fixing the details, I realized that the core logic was missing. I had only wasted precious time that I could have spent figuring it out myself.
[This comment brought to you by ChatGPT - I asked it to write a second draft of my original comment]
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#24GPT-4 is pretty good at generating working Haskell and fixing type errors.
"Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:"
And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell".
But that's not what working programmers do.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#25Earlier quoted context omitted.
> just by telling it that there's a problem how will you identify that there's a problem?
To answer your question from a practical perspective, you can try to run it, and feed back errors. See: https://news.ycombinator.com/item?id=35446171 But that's actually not what I meant. You can often just tell it "there's a problem, please fix it", or "do you see any problems" and it will be able to identify it without additional input. There no requirement that you've identified a problem, it's more of a "double c…
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#26Not to split hairs, but John Henry competed against a steam drill. I'm not sure what an iron track layer is...
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#27GPT-4 is pretty good at generating working Haskell and fixing type errors.
If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#28GPT-4 is pretty good at generating working Haskell and fixing type errors.
If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.
ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format:
{
"items": [
{
"track": {
"album": {
"total_tracks": 9,
"name": "playlist 1",
"genres": [
"Egg punk",
"Noise rock"
]
}
}
},
{
"track": {
"album": {
"total_tracks": 5,
"name": "playlist 2",
"genres": [
"NotEgg punk",
"NotNoise rock"
]
}
}
}
]
}
Here's the code to parse the JSON and print out the album names and genres in CSV format:
```haskell
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson
import Data.List (intercalate)
import GHC.Generics
data Track = Track { album :: Album }
deriving (Show, Generic)
data Album = Album { total_tracks :: Int, name :: String, genres :: [String] }
deriving (Show, Generic)
data Response = Response { items :: [Item] }
deriving (Show, Generic)
data Item = Item { track :: Track }
deriving (Show, Generic)
instance FromJSON Album
instance FromJSON Track
instance FromJSON Item
instance FromJSON Response
main :: IO ()
main = do
json putStrLn "Failed to parse JSON"
Just response -> mapM_ printAlbum $ items response
printAlbum :: Item -> IO ()
printAlbum item = do
let albumName = name . album . track $ item
albumGenres = intercalate ", " $ genres . album . track $ item
putStrLn $ albumName ++ "," ++ albumGenres
```
Assuming the JSON is stored in the file "example.json", running this program will output:
```
playlist 1,Egg punk, Noise rock
playlist 2,NotEgg punk, NotNoise rock
```
This is the album name and genres from each item in the "items" list of the JSON, separated by a comma.
ChatGPT> gpt1.hs:27:23: error:
• Couldn't match type ‘[Char]’
with ‘Data.ByteString.Lazy.Internal.ByteString’
Expected: Data.ByteString.Lazy.Internal.ByteString
Actual: String
• In the first argument of ‘decode’, namely ‘json’
In the expression: decode json :: Maybe Response
In an equation for ‘parsed’: parsed = decode json :: Maybe Response
|
27 | let parsed = decode json :: Maybe Response
|
The error message is indicating that there's a type mismatch between the `json` string and the expected `ByteString` type needed by the `decode` function.
To fix this, you can use the `Data.ByteString.Lazy.Char8` module to create a `ByteString` from the `json` string:
```haskell
import qualified Data.ByteString.Lazy.Char8 as BS
-- ...
main :: IO ()
main = do
json
And it works fine: $ nix-shell -p "haskellPackages.ghcWithPackages (ps: with ps; [aeson])" --run "runhaskell gpt1.hs"
playlist 1,Egg punk, Noise rock
playlist 2,NotEgg punk, NotNoise rockRe: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#29Earlier quoted context omitted.
If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.
Here's a realistic example: ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format: { "items": [ { "track": { "album": { "total_tracks": 9, "name": "playlist 1", "genres": [ "Egg punk", "Noise rock" ] } } }, { "track": { "album": { "total_tracks": 5, "name": "playlist 2", "genres": [ "NotEgg punk", "NotNoise rock" ] } } } ] } Here's the code to parse…
I mean, let's say you really do have to parse this format for something at work.
So you run it on a GB of similar data (meaning not literally specifically a GB, but well more than you can reasonably verify by hand).
Then what? How do you know it works fine, or if it obviously crashes, what to do next?
Re: Benchmarking the accuracy of GPT3.5's and GPT-4's code generation abilities
#30Earlier quoted context omitted.
If I say to ChatGPT, "Write some working Haskell." it replies "Certainly! Here's a simple example of a Haskell program that calculates the factorial of a given number:" And, if I try the code (not knowing Haskell), it appears to work. So I guess it's "good at generating working Haskell". But that's not what working programmers do.
Here's a realistic example: ChatGPT> Parse the album names and genres out of this json using Haskell's Aeson library and print them out in csv format: { "items": [ { "track": { "album": { "total_tracks": 9, "name": "playlist 1", "genres": [ "Egg punk", "Noise rock" ] } } }, { "track": { "album": { "total_tracks": 5, "name": "playlist 2", "genres": [ "NotEgg punk", "NotNoise rock" ] } } } ] } Here's the code to parse…