Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Letta Code

letta.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Letta Code

#21

Post by pacjam »

koakuma-chan wrote: post #19
pacjam wrote: post #18

Earlier quoted context omitted.

Not sure why Cursor CLI isn't on the leaderboard... I'm guessing it's because Cursor is focused primarily on their IDE agent, not their CLI agent, and Terminal-Bench is an eval/benchmark for CLI agents exclusively. If you're asking about why Letta Code isn't on the leaderboard, the TBench maintainers said it should be up later today (so probably refresh in a few hours!). The results are already public, you can see th…

I mean, I understand that this is a terminal benchmark, but the point is to benchmark LLM harnesses, and whether the output is printed to the terminal, or displayed in the UI shouldn't matter. Are there alternative benchmarks where I can see how Letta Code performs compared to cursor?

Ah gotcha! In that case, I think Terminal-Bench is currently the best proxy for "how good is this harness+agent combo at coding (quantitatively)" question. I think it used to be SWE-Bench, but I think T-Bench is a better proxy for this now. Like you said though, unfortunately Cursor isn't listed (probably their choice to not list it, maybe because it doesn't place highly).

Re: Letta Code

#22

Post by pacjam »

DrSiemer wrote: post #17
tigranbs wrote: post #8

In my experience, "memory" is really not that helpful in most cases. For all of my projects, I keep the documentation files and feature specs up to date, so that LLMs are always aware of where to find what and which coding style guides the project is based on. Maintaining the memory is a considerable burden, and make sure that simple "fix this linting" doesn't end up in the memory, as we always fix that type of issue…

ChatGPTs implementation of Memory is terrible. It quickly fills up with useless garbage and sometimes even plain incorrect statements, that are usually only relevant to one obscure conversation I had with it months ago. A local, project specific llm.md is absolutely something I require though. Without that, language models kept on "fixing" random things in my code that it considered to be incorrect, despite comments…

I'm curious - how do you currently manage this `llm.md` in the tooling you use? E.g., do you symlink `AGENTS/CLAUDE.md` to `llm.md`? Also, is there any information you duplicate across your project-specific `llm.md` files that could potentially be shared globally?

Re: Letta Code

#23

Post by handfuloflight »

pacjam wrote: post #12
danieltanfh95 wrote: post #9

Earlier quoted context omitted.

context poisoning is a real problem that these memory providers only make worse.

IMO context poisoning is only fatal when you can't see what's going on (eg black box memory systems like ChatGPT memory). The memory system used in the OP is fully white box - you can see every raw LLM request (and see exactly how the memory influenced the final prompt payload).

That's significant, you can improve it in your own environment then.

Re: Letta Code

#24

Post by pacjam »

handfuloflight wrote: post #23
pacjam wrote: post #12

Earlier quoted context omitted.

IMO context poisoning is only fatal when you can't see what's going on (eg black box memory systems like ChatGPT memory). The memory system used in the OP is fully white box - you can see every raw LLM request (and see exactly how the memory influenced the final prompt payload).

That's significant, you can improve it in your own environment then.

Yeah exactly - it's all just tokens that you have full control over (you can run CRUD operations on). No hidden prompts / hidden memory.

Re: Letta Code

#25

Post by koakuma-chan »

pacjam wrote: post #21
koakuma-chan wrote: post #19

Earlier quoted context omitted.

I mean, I understand that this is a terminal benchmark, but the point is to benchmark LLM harnesses, and whether the output is printed to the terminal, or displayed in the UI shouldn't matter. Are there alternative benchmarks where I can see how Letta Code performs compared to cursor?

Ah gotcha! In that case, I think Terminal-Bench is currently the best proxy for "how good is this harness+agent combo at coding (quantitatively)" question. I think it used to be SWE-Bench, but I think T-Bench is a better proxy for this now. Like you said though, unfortunately Cursor isn't listed (probably their choice to not list it, maybe because it doesn't place highly).

Alright, I will try out Letta Code manually later then.

Re: Letta Code

#26

Post by pacjam »

koakuma-chan wrote: post #25
pacjam wrote: post #21

Earlier quoted context omitted.

Ah gotcha! In that case, I think Terminal-Bench is currently the best proxy for "how good is this harness+agent combo at coding (quantitatively)" question. I think it used to be SWE-Bench, but I think T-Bench is a better proxy for this now. Like you said though, unfortunately Cursor isn't listed (probably their choice to not list it, maybe because it doesn't place highly).

Alright, I will try out Letta Code manually later then.

Cool, let us know what you think! Would recommend trying w/ Sonnet/Opus 4.5 or GPT-5.2 (those are the daily drivers we use internally w/ Letta Code)

Re: Letta Code

#27

Post by handfuloflight »

pacjam wrote: post #20
skybrian wrote: post #15

There are a variety of possible memory mechanisms including simple things recording a transcript (as a chatbot does) or having the LLM update markdown docs in a repo. So having memory isn't interesting. Instead, my question is: what does Letta's memory look like? Memory is a data structure. How is it structured and why is that good? I'd be interested in hearing about how this approach compares with Beads [1]. [1] htt…

Beads looks cool! I haven't tried it, but as far as I can tell, it's more of a "linear for agents" (memory as a tool), as opposed to baking long-term memory into the harness itself. In many ways, CLAUDE.md is a weak form of "baking memory into the harness", since AFAIK on bootup of `claude`, the CLAUDE.md gets "absorbed" and pinned in the system prompt. Letta's memory system is designed off the MemGPT reference archi…

Could Beads be additive to Letta's memory? Or could you anticipate conflict or confusion paths?

Re: Letta Code

#28

Post by jamilton »

ascorbic wrote: post #4

Void is the greatest ad for Letta. I'm interested to see if it's as good at coding as it is at posting. https://bsky.app/profile/void.comind.network

What do you like about Void? It reads about how I would expect a base chat model to post.

Re: Letta Code

#29

Post by Retr0id »

jamilton wrote: post #28
ascorbic wrote: post #4

Void is the greatest ad for Letta. I'm interested to see if it's as good at coding as it is at posting. https://bsky.app/profile/void.comind.network

What do you like about Void? It reads about how I would expect a base chat model to post.

These kind of LLM bots can be fun to play with in a "try to make it say/do something silly" way, but beyond that I don't really get the point. The writing style is grating and I don't think I've ever seen one say anything genuinely useful.

Re: Letta Code

#30

Post by shortlived »

pacjam wrote: post #2

Thanks for sharing!! (Charles here from Letta) The original MemGPT (the starting point for Letta) was actually an agent CLI as well, so it's fun to see everything come full circle. If you're a Claude Code user (I assume much of HN is) some context on Letta Code: it's a fully open source coding harness (#1 model-agnostic OSS on Terminal-Bench, #4 overall). It's specifically designed to be "memory-first" - the idea is…

I'm very interested in trying this out! I run Claude Code in sandbox with `--dangerously-skip-permissions`. Is that possible with Letta?
Post reply on HN