Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An agent in 100 lines of Lisp

thebeach.dev

Post reply on HN
Print view View original

71–80 of 84 posts

Re: An agent in 100 lines of Lisp

#71

Post by Zambyte »

zingar wrote: post #56
goranmoomin wrote: post #10

Earlier quoted context omitted.

> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. > Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generatin…

Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write. Yet I still had the idea that LLMs should be better at lisp than other languages. A fascinating contradiction, thanks for pointing this out.

I believe the problem stems from ) and )) both being one token wide. I wonder if repeated parentheses being removed from the token vocabulary would improve accuracy for Lisp code generation.

Re: An agent in 100 lines of Lisp

#72

Post by cma »

zingar wrote: post #56
goranmoomin wrote: post #10

Earlier quoted context omitted.

> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. > Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generatin…

Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write. Yet I still had the idea that LLMs should be better at lisp than other languages. A fascinating contradiction, thanks for pointing this out.

This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better:

Neural Networks and the Chomsky Hierarchy

https://arxiv.org/abs/2207.02098

Re: An agent in 100 lines of Lisp

#73

Post by killerstorm »

It might be kind of cool to implement just a basic loop and let the agent itself to implement memory and all other aspects on the fly, fully embracing self-modifying code.

But it's also extremely brittle and unstable. It's really the opposite of what people look in agents.

Re: An agent in 100 lines of Lisp

#74

Post by hankbond »

kmeaw wrote: post #16
hankbond wrote: post #2

Maybe this is a reductive comment, but how does this differ from just letting your agent bash tool a `python -c` command (or anything of that class)? I'm not really getting where this is a "wow" moment? It is always nice to appreciate how much power you get out of (Model + the absolute bare minimum of control flow). There is just so much baked into the models now that given an inch they will take a mile.

If you give your agent a `bash` or `python -c` tool, it starts a separate process that produces some output and then exits. After that, only the output and the exit code are available. In contrast, `eval` runs the code in the same execution context as the agent loop. When `eval` finishes, that execution context still exists. For example, any functions defined during an `eval` call remain available for later use.

Ok yeah fair enough I think thats a sizable distinction. FWIW you can do the same thing inside a python process with eval()/exec(), or if we were still keeping it external to the harness process you could just run python as a persistent process listening to stdin. That way you retain the state from when the last command exited.

I wonder if there are any efficiencies to be gained from running a stateful sidecar process like that or if the LLM will have too much of a hard time juggling the state to make coherent followup calls.

Re: An agent in 100 lines of Lisp

#75

Post by drob518 »

derdi wrote: post #49
drob518 wrote: post #47

Earlier quoted context omitted.

Clearly, you’re not going to do it for just Fibonacci. And yes, you can always rewrite the agent code, quit, and restart to modify the agent. This is just like using an LLM to modify Pi and then restarting, which is done today. But being able to rewrite code in a live, running system is useful for all the reasons that Lispers have done it that way for 50+ years. In some cases, you can’t just serialize your state and…

> Sometimes, for instance, you might not even have access to the whole state because some of it resides on a remote server. Not sure what you mean. The system I outlined is one where some "state" resides outside the process in a separate server. You don't need to serialize that, you just need to serialize the information to need to reconnect. And my first point is even more relevant the more complex/distributed/britt…

Cool, you do you.

Re: An agent in 100 lines of Lisp

#76

Post by vichoiglesias »

I always had this idea after reading so much about Lisp that I was designed for AI, but kinda forgot about it with all the craziness of the last years. When I read on the article the eval and the implications of agents self generating their code, it just clicked. Looking forward to experiment with this! kudos to the author!

Re: An agent in 100 lines of Lisp

#77

Post by galaxyLogic »

goranmoomin wrote: post #10
galaxyLogic wrote: post #8

Earlier quoted context omitted.

A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generating co…

> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. > Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generatin…

> JSON is literally a subset of the JavaScript language

Isn't it more like when you write JSON in JavaScript you write it as a String? JavaScript has the data-type "String" but does it also have a built-in data-type "JSON"?

You can pass such a string containing JSON to (built-in) library function JSON.parse() and you can take a data-structure written in JavaScript can call JSON.stringify() on it. But if your JSON is malformed, does the JS-interpreter tell you that, before you call the library-function JUSON.parse() ?

If you have a JavaScript parser/interpreter, does it parse JSON too? If not I would say JSON is not part of the JavaScript language, even if JS runtime provides a library function for parsing it.

Re: An agent in 100 lines of Lisp

#78

Post by galaxyLogic »

zingar wrote: post #55
galaxyLogic wrote: post #8

Earlier quoted context omitted.

A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generating co…

I haven’t seen an LLM generate syntactically invalid code in any language in … a year? So I don’t know if “more likely to be syntactically valid” is a good enough selling point. But maybe I’m interpreting you too narrowly?

I'm thinking that there should be some standard for Agent-LLM communications, and the simpler the better. Lisp might be such a simpler standard as opposed to JSON.

Re: An agent in 100 lines of Lisp

#79

Post by galaxyLogic »

cma wrote: post #72
zingar wrote: post #56

Earlier quoted context omitted.

Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write. Yet I still had the idea that LLMs should be better at lisp than other languages. A fascinating contradiction, thanks for pointing this out.

This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better: Neural Networks and the Chomsky Hierarchy https://arxiv.org/abs/2207.02098

Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
Post reply on HN