Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an interpreter for my programming language with ChatGPT

6502.is-a.dev

Post reply on HN
Print view View original

131–140 of 141 posts

Re: Building an interpreter for my programming language with ChatGPT

#131

Post by EGreg »

samvher wrote: post #87
weatherlite wrote: post #76

Earlier quoted context omitted.

It doesn't sound that much different than going through Google and Stackoverflow though, is it? In a few hours of googling you can probably get something working if you are an experienced dev.

Yeah I had a slightly similar experience as OP, though simpler. I asked it to automate a basic task, something I hadn't done before. I managed to do it with ChatGPT only, no other resources. That said - ChatGPT did make mistakes, there were inconsistencies in its instructions, it didn't recognize certain bugs (I had to find them myself). _But_ there was something about the chat-based interaction that to an extent hel…

ChatGPT helped me close several business deals. I am now a mega millionaire thanks to ChatGPT! Before, I wasn’t able to find the basic info on how to close multi million dollar deals, and I tried all kinds of stuff. But ChatGPT helped me through that. On the calls - whenever I didn’t know what to say next, I would just read off what ChatGPT was responding to the customers, and to my surprise, it matched what they wanted to hear! And they started responding back and forth with it as if it was always in the plan! In the end, they didn’t exactly say “shut up and take my money”, but they did seem to express deep concern that I wouldn’t have availability for them, and essentially agreed to all the upsells very quickly.

I recommend ChatGPT to anyone who wants to close customers or save their marriage. Just say whatever ChatGPT is telling you… even if that means using one of the new “personal” beamed sound into your skull things. You’ll have superhuman ability to vibe with anyone and outcompete everyone who relies on just “their own experience”.

- Written by ChatGPT in response to a prompt.

In the end it added, “no one will ever believe you” in all lowercase.

Re: Building an interpreter for my programming language with ChatGPT

#132

Post by ChristopherDrum »

I don't follow what OP says in the write-up, "I fed it the spec to my language." How? What would that prompt look like? How long did that take? How many prompts? How was it verified that the spec had been understood properly? Etc, etc...

Re: Building an interpreter for my programming language with ChatGPT

#133

Post by skybrian »

dunefox wrote: post #23
mudrockbestgirl wrote: post #9

Earlier quoted context omitted.

The problem is that you cannot trust the output. It's often wrong, but in subtle nonobvious ways. For precise information you still need to check the sources to make sure what you're getting is correct. You can test it out with a (not-so-mainstream) topic that you're an expert in. You'll see lots of mistakes that are obvious to you, but wouldn't be obvious to non-experts. But it's an incredible tool for brainstorming…

So, it's exactly like Google search but more interactive?

Google links to sources that you can read for yourself.

ChatGPT doesn't, so you don't know where it got its information, and you'd have to do web searches anyway to get some possible sources.

Re: Building an interpreter for my programming language with ChatGPT

#135

Post by emrah »

weatherlite wrote: post #76
qnr wrote: post #62

I have been developing a hobby project (AI powered document search) for a few months and was in sore need of a frontend. My frontend development skills however are stuck in late 1990s and I have zero skill with anything but plain HTML and a little bit of JS. Several times I tried learning React, reading tutorials, watching videos, but the whole idea of it was very removed from how I learned to code, so I gave up ever…

It doesn't sound that much different than going through Google and Stackoverflow though, is it? In a few hours of googling you can probably get something working if you are an experienced dev.

One big difference is there is no ads or seo, no one is trying to get in the way of you trying to learn something. Hopefully that will never happen.

Re: Building an interpreter for my programming language with ChatGPT

#136

Post by qayxc »

alchemist1e9 wrote: post #130
qayxc wrote: post #53

Earlier quoted context omitted.

> What is incredible is that it get’s this far. It can compute but not quite correctly yet. That's a conjecture on your part. The ability to compute is quite binary - either it can compute or can't. Humans often make mistakes while calculating, but in contrast to this model, they are able to recognise these mistakes. ChatGPT is incapable of that and often confidentially wrong. My guess is, that there's simply no suit…

It can describe itself well. Here is a dialogue - I want you to act as a python interactive terminal. I will type actions and you will reply with what python would output. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not perform actions unless I instruct you to do so. When I need to tell you something in English I will do so by putting…

Nice, so the model itself confirmed my suspicion that it doesn't do actual calculations.

Rest assured that it won't be able to do that for a while (at least the public versions). Not for technical reasons, though - I'm 100% convinced that the engineers could embed all sorts of interpreters and calculation engines no problem. Just like manual filters and canned responses.

The reason for that is simple: general computation is non-deterministic in both time and space. Inference on the other side is very deterministic in both, because the time taken is a just function of the input length (i.e. current prompt plus a bit of history), which can be well controlled on the frontend.

Arbitrary calculations or code interpretation, however, are unrestricted operations and would basically allow for DoS or straight up breaking the system. While there are ways to limit both calculation time and resource usage, it's still a big risk and considerable effort (infrastructure-wise) that I don't think is worth it. For closed systems (internal rollouts, business customers, etc.) this might be a different story.

Just another reason why closed software sucks: it's not possible for anyone not working at OpenAI to extend the model in ways that would allow for integrations like this to extend its capabilities and test how far we could push it.

Re: Building an interpreter for my programming language with ChatGPT

#137

Post by qayxc »

lordgroff wrote: post #96
qayxc wrote: post #48

Earlier quoted context omitted.

The problem is that the LLM is just that - a language model. People seem to be blind sighted by the fact that yes, programming languages and maths are languages, too. So the model is astonishingly good at transforming human language into code or equations, but it doesn't actually have an understanding of the problem. That's why specialised models such as Codex generate literally tens of millions of solutions and test…

Humans obviously don't "think" the same way. GPT needs memory that humans can't ever have and more importantly an unthinkably large training data set to generate the observations it does. If a human (or another biological system) needed that much training data nothing would have ever gotten off the ground in the first place, it's completely out of reach. This type of a model just doesn't "understand" the same way. St…

> Still, none of this is btw to discount how impressive the technology is. It makes a regular search engine so very quaint by comparison.

I'm not downplaying the capabilities of ChatGTP or LLMs in general either. They're basically a practical implementation of a Chinese Room, which was unthinkable just a few years ago.

What makes it dangerous is the notion to even compare it to a search engine - the two are very different concepts that do very different things. The danger lies in people perceiving the models in exactly that way - a super-powered search engine that they consciously or subconsciously put trust in. The latter is both important and dangerous, because unlike search engines, the output of an LLM cannot be trusted. The model has no concept of differentiating between hallucinated results and extracted knowledge or facts.

At the same time it's capable of generating results in a format that is so convincing that the unsuspecting user cannot easily distinguish made-up output from facts either. This is not an issue that has a near-term technical solution and must be addressed by making users aware of it.

Unlike the 9000 series, ChatGPT is not the most reliable computer ever made. ChatGPT often makes mistakes or distorts information. It is - by any practical definition of the words - not fully approved and very capable of errors.

Re: Building an interpreter for my programming language with ChatGPT

#138

Post by edanm »

dom96 wrote: post #8

I've had a play with ChatGPT and the experience has been pretty frustrating. It either responds with "Sorry, I cannot do this because I don't have access to the internet" (even if I am giving it prompts that don't require this) or it actually generates code but it's subtly incorrect (this was the case when I asked it to generate an example of how to render a 3D cube in JavaScript). This makes me wonder how much time…

> This makes me wonder how much time people are spending optimising the prompt to get the answer they want and they just make it seem like this was the first response they got.

That's sometimes true, but much rarer these days.

Rather, think of it like any skill you have to learn. Someone who doesn't understand much about programming could watch a video of a good programmer writing some code very quickly with awe, and assume the video is a trick in some way. But it's not - the programmer just has enough knowledge and experience that she can do things that other people can't, and do it quickly.

Similarly, if you spent a bit of time working with and learning how to use these models, you can get crazy impressive results every time. You don't have to cherry pick much or at all any more - you just know how to use it properly.

Re: Building an interpreter for my programming language with ChatGPT

#139

Post by throwaway09223 »

lossolo wrote: post #120
throwaway09223 wrote: post #117

Earlier quoted context omitted.

> "if you ask google to show you papers about some topic with words in quotes you think you remember from the paper it will show you the proper link IF it exists and language model will just generate you a result that doesn't exist." Not really. What will actually happen is Google may link me the actual research paper, along with thousands of other associated pages which may or may not have all kinds of fake, false i…

You are mistaking again model with dataset. To be on the same knowledge depth they both have to use the same dataset, the difference is that in search engine you have points of references, rankings and reputation of sites, human discussion and the most important source of the answer, so a lot of signals on which you can also rank the answer yourself. In language model you have none of that, ZERO signals and not only…

> "You are mistaking again model with dataset. "

No, what I'm doing is considering the web content (incentivized by the ad systems Google provides) with the web search systems.

> "and not only it can return answer that was NOT in the dataset but it can make it plausible looking."

This is exactly what SEO spammers do. It's the same.

Re: Building an interpreter for my programming language with ChatGPT

#140

Post by throwaway09223 »

lossolo wrote: post #118
throwaway09223 wrote: post #114

Earlier quoted context omitted.

> "Google is searching in index using keywords and rankings, ChatGPT is predicting plausible words without searching anything anywhere." Expand your scope to both Google, and the creation of an ecosystem of SEO pages which Google incentives. They are the same, in totality. Google doesn't just index -- it also funds the creation of landing pages. > "There is distinction between indexing human knowledge ... and creatin…

> It's a false distinction. Google is more than a search engine; it is also an advertising company that incentivizes original content creation. Obvious straw man argument. Replace word google with search engine. > Expand your scope to both Google, and the creation of an ecosystem of SEO pages which Google incentives. They are the same, in totality. Google doesn't just index -- it also funds the creation of landing pa…

> "Replace word google with search engine."

No, I don't think I will. We are talking about the system as a whole.

> " Search engine will not return to you things that were not in dataset"

Yes it will, because the Google/SEO dynamic is in large part about incentivizing the generation of new content.

I understand you want to narrowly define away this fact to make a point, but the fact stands.

Post reply on HN