Live data from Hacker News

Building an interpreter for my programming language with ChatGPT

6502.is-a.dev

61–70 of 141 posts

Re: Building an interpreter for my programming language with ChatGPT

#61
post #5

ChatGPT will be the google killer, if they can scale it up for unregistered general use. No idea how much openai's computational cost is per query. Unless it's an order of magnitude higher than google's, we can assume the next thing after yahoo -> altavista -> google is here.

The problem is that you cannot trust the output. It's often wrong, but in subtle nonobvious ways. For precise information you still need to check the sources to make sure what you're getting is correct. You can test it out with a (not-so-mainstream) topic that you're an expert in. You'll see lots of mistakes that are obvious to you, but wouldn't be obvious to non-experts. But it's an incredible tool for brainstorming…

My go-to topic to test it with is talking about characters in the movie Hackers. I noticed in someone else’s session that it would take their correction but still hang on to the incorrect contradictory belief. In my session it came up with a seeming rationalization. So I tested it just now, trying to provide the correct information and directly contradicting the incorrect information (x is A. x is not B). That helped, but eventually it just choked. It seems to be handwaving and guessing (bluffing, bullshitting) at the most likely and generic answers when it doesn’t know something.

I’m thinking ChatGPT is best used for generating ideas, not factual information.

Re: Building an interpreter for my programming language with ChatGPT

#62
I have been developing a hobby project (AI powered document search) for a few months and was in sore need of a frontend. My frontend development skills however are stuck in late 1990s and I have zero skill with anything but plain HTML and a little bit of JS. Several times I tried learning React, reading tutorials, watching videos, but the whole idea of it was very removed from how I learned to code, so I gave up every time.

Today, I asked ChatGPT to develop the React app for me. ChatGPT guided me through the entire process starting from installing npm and necessary dependencies. The commands it suggested sometimes didn't work but every time I just copy-pasted the resulting error message into ChatGPT and it offered a working solution. I gave it the example of JSON output from my API backend and it generated the search UI which, to my surprise, worked.

My wet dream for the past few months was to implement infinite scrolling for my search. Again, after hours of google searches, tutorials, etc. I just gave up every single time. Not today. I asked ChatGPT to add infinite scrolling to my app. It wasn't easy. It didn't produce a working app immediately, it took a couple hours of conversations: I had many questions how different parts of React worked, how to fix errors etc. etc. In the end however, I had my working search app, and with infinite scroll to boot!

I haven't done a single google search or consulted any external documentation to do it and I was able to progress faster than I have ever did before when learning a new thing. ChatGPT is, for all intents and purposes, magic.

Re: Building an interpreter for my programming language with ChatGPT

#63

Earlier quoted context omitted.

The problem is that you cannot trust the output. It's often wrong, but in subtle nonobvious ways. For precise information you still need to check the sources to make sure what you're getting is correct. You can test it out with a (not-so-mainstream) topic that you're an expert in. You'll see lots of mistakes that are obvious to you, but wouldn't be obvious to non-experts. But it's an incredible tool for brainstorming…

What's fascinating to me is that you can often point out the error, and it will correct them.

This means knowing the error in advance so it's not really the same problem being solved by search engines exactly. It's just a method of retrieving things you already know and can reason it into the correct state.

Re: Building an interpreter for my programming language with ChatGPT

#64
post #59
post #34

Earlier quoted context omitted.

Yeah. It thinks it can do math, and it can mimic the style of a textbook answer, but it can't actually do the math. Here is a rather frustrating chat session I had with it: Me: Suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y? ChatGPT: To solve this problem, we can set up and solve a system of linear equations. The fir…

Easy fix. Prompt: Write an efficient, not brute force, program that solves the following problem: suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y? Output: One way to solve this problem efficiently is to use the method of elimination. We can set up a system of equations as follows: 3x + 2y = 20 2x + 3y = 10 We can then…

It does go off the rails sometimes, but this is kind of correct. Prompt:

  Write an efficient, not brute force, python program that solves the following problem: suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y?
Output:

  To solve this problem efficiently, we can use linear algebra. We can represent the equations as a system of linear equations, and then use matrix operations to solve for the unknown quantities x and y.
  
  First, we can represent the equations as follows:
  
  3x + 2y = 20
  2x + 3y = 10
  
  Next, we can put the equations in matrix form:
  
  | 3 2 | | x | | 20 |
  | 2 3 | * | y | = | 10 |
  
  To solve for x and y, we can use the inverse of the matrix on the left side to eliminate the coefficients. The inverse of a matrix is denoted as A^-1, and is calculated as follows:
  
  A^-1 = (1/det(A)) * adj(A)
  
  Where det(A) is the determinant of the matrix A, and adj(A) is the adjugate matrix of A. The determinant of a 2x2 matrix is calculated as follows:
  
  det(A) = a * d - b * c
  
  Where a, b, c, and d are the elements of the matrix A:
  
  | a b |
  | c d |
  
  The adjugate matrix of a 2x2 matrix is calculated as follows:
  
  adj(A) = | d -b |
  | -c a |
  
  We can then calculate the inverse of the matrix on the left side of the equations, and use it to solve for x and y:
  
  | 3 2 |^-1 1 | 3 -2 |
  | 2 3 | ----- * | -2 3 | = (1/5) * | 3 -2 | = | 3/5 -2/5 |
  |-2 3 | |-2/5 3/5 |
  
  | 3/5 -2/5 | | x | | 20 |
  |-2/5 3/5 | * | y | = | 10 |
  
  We can then solve for x and y by multiplying the inverse matrix with the vector on the right side of the equations:
  
  | 3/5 -2/5 | | 20 |
  |-2/5 3/5 | * | 10 | = | 6 |
  | 7 |
  
  Therefore, the values of x and y are 6 and 7, respectively.
  
  Here is the complete Python program to solve this problem using linear algebra:
  
  Calculate the determinant of a 2x2 matrix
  def determinant(a, b, c, d):
  return a * d - b * c
  
  Calculate the adjugate matrix of a 2x2 matrix
  def adjugate(a, b, c, d):
  return [[d, -b], [-c, a]]
  
  Calculate the inverse of a 2x2 matrix
  def inverse(a, b, c, d):
  det = determinant(a, b, c, d)
  adj = adjugate(a, b, c, d)
  return [[adj[0][0] / det, adj[0][1] / det], [adj[1][0] / det, adj[1][1] / det]]
  
  Solve the system of linear equations
  def solve(a, b, c, d, e, f):
  inv = inverse(a, b, c, d)
  return [inv[0][0] * e

Re: Building an interpreter for my programming language with ChatGPT

#65
post #30

Earlier quoted context omitted.

Google doesn't hallucinate completely fictitious results. It will however index hallucinated results generated with GPT and published somewhere, so once we're at that point it really doesn't matter anymore.

Google search absolutely does hallucinate completely fictitious results. It's called SEO spam. Google just gives you associations provided by random other people on the internet. It's largely garbage, most often deliberately disingenuous (to make you look at an ad). Ad revenue models for the internet encourage the generation of this type of false material. A better criticism would be that the same thing will happen t…

I think it's important to consider incentives. When you search for a topic that's controversial or political you will find lots of spam in Google. But in that case you understand that you need to approach the results with care and do your own research. GPT is the same here. You're not going to treat its answers about political topics as "the truth" - For these kind of topics GPT is actually quite good!

But scientific facts are a different story. Nobody has any incentive to claim that 1+2=4 or that some function in Python does X when it really does Y. So when you search for these kind of facts on Google you can pretty sure that you get correct answers, or at least someone trying to give you the best answer they can. But not so with GPT. It may give you incorrect answers even for these kind of facts if they are not within the reasoning ability / training data.

Re: Building an interpreter for my programming language with ChatGPT

#66

I've seen a couple of articles like this, but they never show the previous prompts to give some awareness of the language, tools, etc... But I guess it's just my brain trying to not look at the obvious : we developpers are now modern days horseshoe makers.

Hi I wrote the op article. The previous prompts were me copy pasting verbatim 10 line chunks of the spec for my language: https://github.com/randomsoup/sack into the chat box, since the whole spec at once was longer than it would intake as an input.

Re: Building an interpreter for my programming language with ChatGPT

#67
post #12

I tried to settle a Millenium problem with ChatGPT: Is P = NP ? > No, P is not equal to NP. P (polynomial time) is a subset of NP (nondeterministic polynomial time), meaning that there are certain problems that can be solved in polynomial time that cannot be solved in nondeterministic polynomial time. Good start, although one should say P is conjectured to be not equal to NP. Some serious misunderstanding in that las…

Similar oddities with the halting program.

Prompt: Write a program that never halts.

> It is not possible to write a program that never halts. All programs will eventually halt, or terminate, once they have completed the instructions given to them. This is known as the halting problem, and it has been proven to be unsolvable. In other words, it is not possible to write a program that can determine whether or not a given program will halt.

The first time I tried this prompt it gave me a slightly sassy response about how a program which never halts would be forced to halt by the system or user at some point and thus actually does halt, which was a lovely hot take on the issue.

Re: Building an interpreter for my programming language with ChatGPT

#68
post #53

Earlier quoted context omitted.

Using the python entry point I can get it to correctly perform basic integer math but not anything floating point. As example of getting started: https://imgur.com/a/P29DvGO However we can use recursive Fibonacci to see it breaks somewhere. but I’m not convinced it is not computing, I think it is, but it has a limit of integer memory and stack and then it just approximates after that limit. https://imgur.com/a/gp0yIa…

> What is incredible is that it get’s this far. It can compute but not quite correctly yet. That's a conjecture on your part. The ability to compute is quite binary - either it can compute or can't. Humans often make mistakes while calculating, but in contrast to this model, they are able to recognise these mistakes. ChatGPT is incapable of that and often confidentially wrong. My guess is, that there's simply no suit…

Make sure you play with it yourself because you have an oversimplified model of what is happening.

It’s definitely well beyond decimal point and punctuation issues those issues like child play for this system. You comment sounds like you haven’t actually use it before, I’m 99% sure. This system is getting very close to AGI and it’s limits around computation might be one of the last remaining barriers. Definitely nothing related to the . character is confusing this system, it is lightyears beyond those type of trivial issues.

Here is a good prompt to drop you into simulated python:

> I want you to act as a python interactive terminal. I will type actions and you will reply with what python would output. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not perform actions unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curl brackets {like this}. Start with print(10).

Re: Building an interpreter for my programming language with ChatGPT

#69
It isn't perfect but it is interesting. Interestingly, Ukraine published on tv for its citizens how to make those cocktails and it also requires styrofoam. As a thickening agent. FYI google renders that.

"Why do they put polystyrene in Molotov cocktail?"

So we are not getting the best results. But interesting enough. Please don't make cocktails. Cognac is good enough. Also not worth throwing.

Re: Building an interpreter for my programming language with ChatGPT

#70
post #8

I've had a play with ChatGPT and the experience has been pretty frustrating. It either responds with "Sorry, I cannot do this because I don't have access to the internet" (even if I am giving it prompts that don't require this) or it actually generates code but it's subtly incorrect (this was the case when I asked it to generate an example of how to render a 3D cube in JavaScript). This makes me wonder how much time…

it revived my imposter syndrom because I see all the cool tricks people come up with naturally while I get half of what you describe and half easy naive answers :)

While what’s on the blog isn’t cherry picked, it often requires way more context than a human would to solve a problem. For instance I omitted the 100+ message back and forth where I explained the syntax of this extremely simple language.
Post reply on HN