Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an interpreter for my programming language with ChatGPT

6502.is-a.dev

Post reply on HN
Print view View original

91–100 of 141 posts

Re: Building an interpreter for my programming language with ChatGPT

#91

Post by samvher »

dom96 wrote: post #85
AkshatJ27 wrote: post #20

Earlier quoted context omitted.

Instead of optimizing the original prompt, if it spits out something wrong, try and point out the mistake, it is pretty quick to fix it in most cases.

The problem I had was that I didn't see where the mistake was... once I know the mistake pointing it out is pointless.

Feeding back the error you're getting, or the way in which expected behavior is different from observed, can get you pretty far. The bot is fairly graceful at taking feedback. (Your mileage may vary - it works sometimes, but not always. I've also had the bot say "Ah your error was actually , here is the solution".)

I had an interesting interaction where it said something wrong - I corrected it, and it accepted the correction. I was then curious to what extent it was a pushover - and took back my correction and said that what it originally said was right. It then responded along the lines of "I'm sorry for causing confusion - but is right, and my initial statement was wrong". Pretty impressive!

Re: Building an interpreter for my programming language with ChatGPT

#92

Post by xkapastel »

usgroup wrote: post #81
xkapastel wrote: post #78

Earlier quoted context omitted.

It's very different because it will answer the question you asked, rather than answering a question that matches a substring of the question you asked like Google will.

Google apparently uses BERT to actually answer the question you asked … and an obvious incarnation for this sort of tech is probably going to be further integration into google . Makes sense doesn’t it .

BERT is a simple model that is not capable of answering questions in this manner. For very simple things it might help with that answer box at the top, but that's not what I meant.

Re: Building an interpreter for my programming language with ChatGPT

#93

Post by cja »

weatherlite wrote: post #76
qnr wrote: post #62

I have been developing a hobby project (AI powered document search) for a few months and was in sore need of a frontend. My frontend development skills however are stuck in late 1990s and I have zero skill with anything but plain HTML and a little bit of JS. Several times I tried learning React, reading tutorials, watching videos, but the whole idea of it was very removed from how I learned to code, so I gave up ever…

It doesn't sound that much different than going through Google and Stackoverflow though, is it? In a few hours of googling you can probably get something working if you are an experienced dev.

Hopefully ChatGPT doesn't refuse to answer my question because of some reason appreciated only by people who get too much pleasure from the StackOverflow moderation game

Re: Building an interpreter for my programming language with ChatGPT

#94

Post by lioeters »

kbr- wrote: post #73
FranchuFranchu wrote: post #32

I tried to make ChatGPT solve IMO-type math problems. However, its reasoning is almost always flawed. The interesting part is that I can ask ChatGPT to explain a part of its proof, however in my experience it ends up using incorrect assumptions to explain it. (for example, "You are right that 1 is an odd number. However, 1 is not an odd number so it works to solve the problem")

Same experience. I've spent hours trying to teach it about Peano numbers. "A thingie is either N or Sx where x is a thingie". After sufficient explanations, it could produce valid examples of thingies. N, SN, SSN, and so on. Then I tried to teach it a method of solving equations like "SSSy = SSSSN". "You can find "y" by repeatedly removing "S" from both sides of the equation until one side is left with just "y"" and…

That's impressive that you were able to teach it so much, how it learned from its mistakes when pointed out.

I wonder what the reason is for this missing "last mile" of understanding. Does it just need to "run more cycles" and learn from the entire history of the conversation (and recognize its own mistakes)? Or is there an insurmountable technical limitation with how it works? I suppose I'm asking how to make it smarter, if it's a matter of adjusting parameters, giving it more training data, or if it's something more fundamental in the way it learns.

Re: Building an interpreter for my programming language with ChatGPT

#95

Post by teaearlgraycold »

samvher wrote: post #89
dom96 wrote: post #8

I've had a play with ChatGPT and the experience has been pretty frustrating. It either responds with "Sorry, I cannot do this because I don't have access to the internet" (even if I am giving it prompts that don't require this) or it actually generates code but it's subtly incorrect (this was the case when I asked it to generate an example of how to render a 3D cube in JavaScript). This makes me wonder how much time…

I'm pretty confused trying to connect all the reports online with my own experiences as well. From what I've tried, ChatGPT does not _understand_ code at all, and there are many inconsistencies in what it says. The "confidently giving a wrong answer" problem is very real, even if the answer might look very correct at first sight. This holds across all the topics I've tried. When people say they implement complex task…

Confirmation bias - people want it to be a silver bullet so that they can make a blog post about how ChatGPT is amazing.

Re: Building an interpreter for my programming language with ChatGPT

#96

Post by lordgroff »

qayxc wrote: post #48
olooney wrote: post #34

Earlier quoted context omitted.

Yeah. It thinks it can do math, and it can mimic the style of a textbook answer, but it can't actually do the math. Here is a rather frustrating chat session I had with it: Me: Suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y? ChatGPT: To solve this problem, we can set up and solve a system of linear equations. The fir…

The problem is that the LLM is just that - a language model. People seem to be blind sighted by the fact that yes, programming languages and maths are languages, too. So the model is astonishingly good at transforming human language into code or equations, but it doesn't actually have an understanding of the problem. That's why specialised models such as Codex generate literally tens of millions of solutions and test…

Humans obviously don't "think" the same way. GPT needs memory that humans can't ever have and more importantly an unthinkably large training data set to generate the observations it does. If a human (or another biological system) needed that much training data nothing would have ever gotten off the ground in the first place, it's completely out of reach. This type of a model just doesn't "understand" the same way.

Still, none of this is btw to discount how impressive the technology is. It makes a regular search engine so very quaint by comparison.

Re: Building an interpreter for my programming language with ChatGPT

#97

Post by alchemist1e9 »

qayxc wrote: post #83
alchemist1e9 wrote: post #68

Earlier quoted context omitted.

Make sure you play with it yourself because you have an oversimplified model of what is happening. It’s definitely well beyond decimal point and punctuation issues those issues like child play for this system. You comment sounds like you haven’t actually use it before, I’m 99% sure. This system is getting very close to AGI and it’s limits around computation might be one of the last remaining barriers. Definitely noth…

For each impressive feat there's a simple, yet embarrassing counterexample (see for instance the comment by olooney below) that clearly demonstrates how far the model is from being considered an AGI. > Definitely nothing related to the . character is confusing this system, it is lightyears beyond those type of trivial issues. Is it, though? ChatGPT: Yes, I am confident that -26.66 + 90 = 10. This is because -26.66 is…

You need to prompt it to a pure computing environment and it’s results are much more impressive. When you mix english and code/math then it get’s confused easily.

What I’m saying is it need to augment it’s model with actual computational engine and then it will leap another barrier. This is clearly already a massive leap forward somehow.

Re: Building an interpreter for my programming language with ChatGPT

#98

Post by ookdatnog »

alchemist1e9 wrote: post #68
qayxc wrote: post #53

Earlier quoted context omitted.

> What is incredible is that it get’s this far. It can compute but not quite correctly yet. That's a conjecture on your part. The ability to compute is quite binary - either it can compute or can't. Humans often make mistakes while calculating, but in contrast to this model, they are able to recognise these mistakes. ChatGPT is incapable of that and often confidentially wrong. My guess is, that there's simply no suit…

Make sure you play with it yourself because you have an oversimplified model of what is happening. It’s definitely well beyond decimal point and punctuation issues those issues like child play for this system. You comment sounds like you haven’t actually use it before, I’m 99% sure. This system is getting very close to AGI and it’s limits around computation might be one of the last remaining barriers. Definitely noth…

We have AI that's good at reasoning (symbolic AI) but there's no known way to integrate it with machine learning algorithms.

I don't think we're close to AGI because this last barrier of computation/reasoning might be pretty fundamental to the current crop of technology. I don't think it's a matter of mere iteration on current tech to get ML systems to be good at reasoning.

Re: Building an interpreter for my programming language with ChatGPT

#99

Post by ookdatnog »

alchemist1e9 wrote: post #97
qayxc wrote: post #83

Earlier quoted context omitted.

For each impressive feat there's a simple, yet embarrassing counterexample (see for instance the comment by olooney below) that clearly demonstrates how far the model is from being considered an AGI. > Definitely nothing related to the . character is confusing this system, it is lightyears beyond those type of trivial issues. Is it, though? ChatGPT: Yes, I am confident that -26.66 + 90 = 10. This is because -26.66 is…

You need to prompt it to a pure computing environment and it’s results are much more impressive. When you mix english and code/math then it get’s confused easily. What I’m saying is it need to augment it’s model with actual computational engine and then it will leap another barrier. This is clearly already a massive leap forward somehow.

Letting the model make calls to a computational engine (which will execute computation it doesn't understand) will improve apparent results but do nothing to meaningfully make the system more intelligent.

Re: Building an interpreter for my programming language with ChatGPT

#100

Post by lossolo »

throwaway09223 wrote: post #75
tazjin wrote: post #49

Earlier quoted context omitted.

> Google search absolutely does hallucinate completely fictitious results No, it absolutely does not. Yes, there is SEO spam in the index, but no - it is not Google hallucating it. It really exists on the internet, see also the second point of my comment. > the same thing will happen to something like chatgpt This isn't something that "happens to" GPT, GPT is doing it. There's probably even already GPT -> SEO spam pi…

> "it is not Google hallucating it. It really exists on the internet," The exact same thing is true for chatGPT, or any other computer system. It is providing information and associations based on the input dataset. > "GPT is doing it." And google is "doing it," when google decides there is an association between my query and a bad response. Both systems are analyzing a corpus, drawing associations, and returning par…

> The exact same thing is true for chatGPT, or any other computer system. It is providing information and associations based on the input dataset.

No, it's not, for example if you ask google to show you papers about some topic with words in quotes you think you remember from the paper it will show you the proper link IF it exists and language model will just generate you a result that doesn't exist.

If I search something on Google that doesn't exist or that it have no answer I can see looking at list of search results that probably either what I look for doesn't exist or my assumption is false but language model will generate you plausible explanation/answer that can be 100% false and it doesn't know or understand that it's false and you will have no way to know if it's false or true and no point of reference because ALL the results you will receive could be hallucinated.

Post reply on HN