Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TERMy – A fast terminal assistant that does not use LLMs

github.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#11

Post by kouteiheika »

gurjeet wrote: post #6
gioscarab wrote: post #2

Hi, I am the creator, feel free to ask any questions :) What do you think about it?

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design. TERMy (or is it the NPC-forge) seems to be worth a try.

> because their output is non-deterministic by design.

It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#12

Post by gioscarab »

kouteiheika wrote: post #10
gioscarab wrote: post #2

Hi, I am the creator, feel free to ask any questions :) What do you think about it?

> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM. Have you considered/tried using a model that's, well, more appropriate size-wise for an use case like this? These are relatively big. Something like FunctionGemma [1] finetuned for a given set of tasks would be a lot more speedy. [1] https…

I tried functiongemma, it is for sure faster than those models, the problem is that is not reliable enough for a terminal assistant. I would say that no LLM is good for a terminal assistant, if you take into account the operational cost and the risk of damage. Even if it fails only 1 time out of 10 becomes useless. That's why I developed FlintParser!

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#13

Post by [deleted] »

gioscarab wrote: post #9
utopiah wrote: post #8

Earlier quoted context omitted.

What dataset does step 5 rely on? Is it from your own terminal history, man pages, scrapped dataset from e.g. StackOverflow, sth else?

The dataset is here: https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy/dat... I hope the community will help me to enhance it :) it is just a proof of concept for now

[deleted]

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#14

Post by coder543 »

kouteiheika wrote: post #10
gioscarab wrote: post #2

Hi, I am the creator, feel free to ask any questions :) What do you think about it?

> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM. Have you considered/tried using a model that's, well, more appropriate size-wise for an use case like this? These are relatively big. Something like FunctionGemma [1] finetuned for a given set of tasks would be a lot more speedy. [1] https…

FunctionGemma never worked well for me (without fine tuning). Liquid has released 230M and 350M models that work far, far better in my testing: https://huggingface.co/LiquidAI/LFM2.5-230M

I really look forward to a hypothetical LFM3-230M, because LFM2.5-230M is so close to being usable, while FunctionGemma is miles away from being usable.

But, yes, still tangential to TERMy.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#16

Post by gioscarab »

indigodaddy wrote: post #15

So is this kind of like a super-powered tealdeer ?

tealdeer just shows you a cheatsheet, termy can effectively take a prompt and execute a command, example:

$ termy create file test.txt and write Hello

TERMy | template match | Confidence: 100.00%

Thinking: Ok, I am asked to create the file test.txt.

echo 'Hello' > 'test.txt' && termy_set_context 'active_file' 'test.txt'

Description: Writes Hello in file test.txt.

Response: Affirmative

Now that I think about it, I should let TERMy use tldr...

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#17

Post by piterrro »

gurjeet wrote: post #6
gioscarab wrote: post #2

Hi, I am the creator, feel free to ask any questions :) What do you think about it?

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design. TERMy (or is it the NPC-forge) seems to be worth a try.

You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#18

Post by kennywinker »

kouteiheika wrote: post #11
gurjeet wrote: post #6

Earlier quoted context omitted.

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design. TERMy (or is it the NPC-forge) seems to be worth a try.

> because their output is non-deterministic by design. It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response to very similar prompts.

This tool has a finite amount of outputs for an infinite amount of inputs. Which is different from an llm based tool.

Re: Show HN: TERMy – A fast terminal assistant that does not use LLMs

#20

Post by kennywinker »

kouteiheika wrote: post #10
gioscarab wrote: post #2

Hi, I am the creator, feel free to ask any questions :) What do you think about it?

> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM. Have you considered/tried using a model that's, well, more appropriate size-wise for an use case like this? These are relatively big. Something like FunctionGemma [1] finetuned for a given set of tasks would be a lot more speedy. [1] https…

https://github.com/ThorOdinson246/whatisit-nl2sh uses a finetune of Qwen2.5-Coder-1.5B-Instruct. It works pretty well, tho it will misunderstand things from time to time
Post reply on HN