Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TERMy – A fast terminal assistant that does not use LLMs

github.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#21

Post by jdiff »

piterrro wrote: post #17
gurjeet wrote: post #6

Earlier quoted context omitted.

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design. TERMy (or is it the NPC-forge) seems to be worth a try.

You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.

That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#23

Post by skeledrew »

kennywinker wrote: post #18
kouteiheika wrote: post #11

Earlier quoted context omitted.

> because their output is non-deterministic by design. It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response t…

I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#24

Post by asQuirreL »

jdiff wrote: post #21
piterrro wrote: post #17

Earlier quoted context omitted.

You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.

That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.

That's the definition of a chaotic system (small change in initial conditions results in large, seemingly -- but not actually -- random changes in output), but it's still deterministic (same input results in same output).

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#26

Post by gioscarab »

mbil wrote: post #25

It's kind of antithetical to the tool's deterministic positioning, but have you considered making TERMy leverage an LLM for unseen or low-confidence queries, and then generate the config and update itself to make future similar queries deterministic?

This is such a nice idea! I could add a fallback towards LLMs, it was present but I removed it. Would you be interested to help me implement the auto-update? I must admit, the LLMs are very useful for this kind of work. I think that TERMY's design is now feasible BECAUSE OF the availability of LLMs. They make the dataset development feasible.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#27

Post by dmos62 »

It would make sense to have this integrate with a self-learning routine for an agent: e.g. at night it looks through what it did and writes NPC-Forge recipes. Tomorrow it can answer queries (which he turned added to NPC-Forge) without an LLM. Of course this implies a branching where a query is either processed by NPC-Forge or an LLM, depending on some measure of confidence that NPC-Forge can answer it well.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#28

Post by kennywinker »

skeledrew wrote: post #23
kennywinker wrote: post #18

Earlier quoted context omitted.

That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response t…

I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.

Yes i think I acknowledged that, but is that useful for making a tool that can be trusted to safely run shell commands when asked arbitrary questions? No. It’s not.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#29

Post by gioscarab »

dmos62 wrote: post #27

It would make sense to have this integrate with a self-learning routine for an agent: e.g. at night it looks through what it did and writes NPC-Forge recipes. Tomorrow it can answer queries (which he turned added to NPC-Forge) without an LLM. Of course this implies a branching where a query is either processed by NPC-Forge or an LLM, depending on some measure of confidence that NPC-Forge can answer it well.

Yeah would be a nice experiment, if you are interested to contribute to NPC-Forge please open an issue, I would be happy to discuss about that.
Post reply on HN