Small offline large language model – TinyChatEngine from MIT
graphthinking.blogspot.com
Small offline large language model – TinyChatEngine from MIT
1–10 of 25 posts
Re: Small offline large language model – TinyChatEngine from MIT
#2I’m a tad confused
> TinyChatEngine provides an off-line open-source large language model (LLM) that has been reduced in size.
But then they download the models from huggingface. I don’t understand how these are smaller? Or do they modify them locally?
Re: Small offline large language model – TinyChatEngine from MIT
#3I’m a tad confused > TinyChatEngine provides an off-line open-source large language model (LLM) that has been reduced in size. But then they download the models from huggingface. I don’t understand how these are smaller? Or do they modify them locally?
https://github.com/mit-han-lab/TinyChatEngine
Turns out the original source is actually somewhat informative. Including telling you how much hardware do you need. This blog post looks like your typical note you leave for yourself to annotate a bit of your shell history.
Re: Small offline large language model – TinyChatEngine from MIT
#4I’m a tad confused > TinyChatEngine provides an off-line open-source large language model (LLM) that has been reduced in size. But then they download the models from huggingface. I don’t understand how these are smaller? Or do they modify them locally?
https://github.com/mit-han-lab/TinyChatEngine Turns out the original source is actually somewhat informative. Including telling you how much hardware do you need. This blog post looks like your typical note you leave for yourself to annotate a bit of your shell history.
I wish that all these repos were more clear about the hardware requirements. Seeing that it runs on a 8 GB Raspberry, probably with abysmal performance, I'd say that it will run on my 32 GB Intel laptop on the CPU. Will it run on its Nvidia card? I remember that the rule of thumb was one GB of GPU RAM per G parameters, so I'd say that it won't run. However this has 4 bit quantization so it could have lower requirements.
Of course the main problem is that I don't know enough about the subject to reason on it on my own.
Re: Small offline large language model – TinyChatEngine from MIT
#5I have used them and I can say it's pretty decent overall. I personally plan to use tinyengineon iot devices which is for even smaller iot microcontroller devices.
Re: Small offline large language model – TinyChatEngine from MIT
#6Use llama.cpp for quantized model inference. It is simpler (no Docker nor Python required), faster (works well on CPUs), and supports many models.
Also there are better models than the one suggested. Mistral for 7B parameters. Yi if you want to go larger and happen to have 32Gb of memory. Mixtral MoE is the best but requires too much memory right now for most users.
Re: Small offline large language model – TinyChatEngine from MIT
#7Earlier quoted context omitted.
https://github.com/mit-han-lab/TinyChatEngine Turns out the original source is actually somewhat informative. Including telling you how much hardware do you need. This blog post looks like your typical note you leave for yourself to annotate a bit of your shell history.
I wish that all these repos were more clear about the hardware requirements. Seeing that it runs on a 8 GB Raspberry, probably with abysmal performance, I'd say that it will run on my 32 GB Intel laptop on the CPU. Will it run on its Nvidia card? I remember that the rule of thumb was one GB of GPU RAM per G parameters, so I'd say that it won't run. However this has 4 bit quantization so it could have lower requiremen…
Roughly speaking I believe it's the number of parameters times the size of the parameters. So in the 4 bit case it's half a gigabyte per billion parameters.
From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.
Re: Small offline large language model – TinyChatEngine from MIT
#8I have used them and I can say it's pretty decent overall. I personally plan to use tinyengineon iot devices which is for even smaller iot microcontroller devices.
May I ask what your use case is? I've found LLMs are pretty good at parsing unstructured data into JSON, with minimal hallucinations.
Re: Small offline large language model – TinyChatEngine from MIT
#9"Small large" ..... so, medium? :)
Re: Small offline large language model – TinyChatEngine from MIT
#10I’m a tad confused > TinyChatEngine provides an off-line open-source large language model (LLM) that has been reduced in size. But then they download the models from huggingface. I don’t understand how these are smaller? Or do they modify them locally?
They have postprocessed the models specifically for size and latency. They published several papers on this.
Their optimized models are not downloaded from HF, but from dropbox. I have no idea why.