[dead]
Is anyone using self hosted LLM day to day and training it like a new employee
41–50 of 75 posts
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#42Ask HN:
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#43The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
I would imagine that daily "training" here involves something more like RLHF than just appending to a big prompt.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#44Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF But yeah current context windows are limiting.
With RAG and fine tuning (which is cheap), you can fine tune a model on a daily basis so that one isn't trying to stuff everything in the context window.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#45The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.
The animal brain equivalent isn't summarize a context window to account for limited working memory. It's to never leave training mode to go into inference-only mode. The learned models in animal brains never stop learning.
There is nothing stopping someone from keeping an LLM in online-training mode forever. We don't do that because it's economically infeasible, not because it wouldn't work.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#46Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time. If your goal is to bake new concepts into the model weights, your only real option is a dataset with tha…
Yeah, I think training for facts in general is kind of problematic since you often have to overfit and the model may lose capability in other areas. I suspect that the only situations where it really makes sense to train on facts are where the facts are very nuanced and require a lot of interpretation, or more often where the facts are just so extensive that they can't be crammed effectively into the context window y…
What if we just train it to respect facts in general, then couldn't we just supply it a list of facts?
Sort of how they made chatGPT way more likely to obey requests?
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#47Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF But yeah current context windows are limiting.
I was testing this one days ago. It seems fine to use as a base for extra finetuning, but failed hard questions that chatgpt nailed. One example was trying to use as a assistant to beat long games, without immediate rewards. I was trying to log and simultaneously get feedback playing Stardew Valley. gpt-3.5-turbo-1106 basically went along with me and my daughter in a coop session giving nice suggestions, sometimes wi…
True, and makes sense that the logic is closing in but the breadth of the data is too narrow in 7GB's to ask questions about niche topics.
Mistral hasn't released their own official 13B/30B's yet, but i'm really looking forward to what they can do.
What is crazy is that Ultrafastbert, Speculative, Jacobi, or lookahead decoding could potentially speed up by up to 80x depending on size which could make GPT-4 like models feasible on entry level macs / Phones if similar wizardry is done memory wise.
..Yes im very optimistic after the insane progress over the last months with models like Mistral, Deepseek etc.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#48I would like to extend the question: is anyone building a homelab for the specific purpose of training a LLM on their personal info? The choice of hardware (for speed, cost, and noise concerns) seems important.
For me, nothing fancy, I just added extra ram to a gaming notebook to get enough speed on answers, since it already as a good nvidia card, keeping the api open for access from another laptop I have, via api inside my local network.
I have an extra computer for services like filesharing, samba, nfs, git, firewall, etc, for instance caching the models I'm downloading with a squid proxy, so I can test several UIs downloading the same model over again. Not every UI is offering an easy way to set a single folder to store all gguf files, or it's lacking documentation.
I'm already having a lot of fun. There's people already doing much more than this. I'm more worried about integrating and gluing in a way that will become transparent after the new year, local models or not.
Also how to glue this with obsidian/logseq/neovim/etc in a way that I can use with fewest keystrokes possible, instead just uploading a gigantic context or sensible source code files.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#49Earlier quoted context omitted.
Yeah, I think training for facts in general is kind of problematic since you often have to overfit and the model may lose capability in other areas. I suspect that the only situations where it really makes sense to train on facts are where the facts are very nuanced and require a lot of interpretation, or more often where the facts are just so extensive that they can't be crammed effectively into the context window y…
What if we just train it to respect facts in general, then couldn't we just supply it a list of facts? Sort of how they made chatGPT way more likely to obey requests?
You can supply the model with a list of facts already, that’s not the problem. Within the context window the model is able to learn and generalize new information.
Fine tuning is very unintelligent in the sense that it doesn’t take the context of the training samples into account. It’s a dumb optimizer that’s trying to minimize next token loss. Gradient descent is not beholden to the behaviors you taught in the instruct fine tune step.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#50The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
The solution is RAG