Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is anyone using self hosted LLM day to day and training it like a new employee

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#31

Post by TrevorJ »

MyFirstSass wrote: post #23

Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF But yeah current context windows are limiting.

I think with RAG it's pretty reasonable. Put your corpus in pinecone or some other vector store and relevant sections get injected along with your prompt which lessens the burden on context window.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#32

Post by hushpiper »

valine wrote: post #7

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time. If your goal is to bake new concepts into the model weights, your only real option is a dataset with tha…

Yeah, I think training for facts in general is kind of problematic since you often have to overfit and the model may lose capability in other areas. I suspect that the only situations where it really makes sense to train on facts are where the facts are very nuanced and require a lot of interpretation, or more often where the facts are just so extensive that they can't be crammed effectively into the context window you're working with. Otherwise, you're better off with a vector db and a well-written prompt.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#33

Post by xtracto »

I was thinking on doing something among the same lines but with code:

An LLM that is specifically trained for Software Development, and to which I feed the code of all my company's repositories. And I keep feeding commits/pull requests.

The idea is that I can query it about architectural issues, code improvements, and other technical aspects at different levels of abstraction (code, architecture, business, etc).

So far, I've played a bit with CodeRabbit and it's "just ok" but it is more of a very small windows to what "could be" than being actually useful.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#34

Post by lukebuehler »

adammichaelc wrote: post #19
abstrct wrote: post #11

Earlier quoted context omitted.

I’ve absolutely explored this idea but, similar to lossy compression, sometimes important nuance is lost in the process. There is both an art and science to recalling the gently compacted information and being able to recognize when it needs to be repeated back.

If there was something like Objects in OO programming, but for LLM’s, would that solve this? Like a Topic-based Personality Construct where the model first determines which of its “selves” should answer the question, and then grabs appropriate context given the situation.

look up "frames", it's an old concept and also influenced OOP.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#35

Post by LegitShady »

mtlynch wrote: post #22
semireg wrote: post #8

As a solo developer answering emails that basically point people to various guides and FAQs I’ve published … I need this. Zendesk claims to have an AI component but forces you to input all training data into their own wiki knowledge base. I can see why they don’t want to use prior responses as training (pii concerns), but at least give me some boilerplate responses that I can use to get a head start and further train…

Why not do it the old fashioned way and hire a human for this? Humans also have the advantage that they don't just make up answers when they don't know something (or at least if you hire good ones). I've had good experience hiring support folks and working with them on a shared inbox (we use HelpScout).

he is the human that does this.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#36

Post by jacquesm »

ijwann wrote: post #27
abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

> Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it. Seems very realistic!

No, it would be realistic if after two weeks on the job they start telling you how to run the company.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#37

Post by kqr »

abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

What about some generation-augmented retrieval augmented generation set-up where all your conversations are indexed for regular text search, and then you use the LLMs language knowledge to generate relevant search phrases the results of which are included in the current prompt?

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#38

Post by TrevorJ »

unoti wrote: post #25
TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

> What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today this might be a good time". That sort of thing. If you work in a Microsoft world, this is what GraphAPI is all about: enabling access to all the things using your personal authentication to…

interesting, thank you.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#39

Post by rodrigodlu »

MyFirstSass wrote: post #23

Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop? Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF But yeah current context windows are limiting.

I was testing this one days ago. It seems fine to use as a base for extra finetuning, but failed hard questions that chatgpt nailed.

One example was trying to use as a assistant to beat long games, without immediate rewards.

I was trying to log and simultaneously get feedback playing Stardew Valley. gpt-3.5-turbo-1106 basically went along with me and my daughter in a coop session giving nice suggestions, sometimes with huge gaps, but easy enough to ask more about after giving more context.

Mistral 7b and 13B was basically mixing up stardew valley with WoW and Genshin Impact, even giving a lot of context about the day I was, what the npcs answered, or things that I know on how to solve a certain quest. It straight made up non existing towns (stardew valley only has one) etc, etc.

I was running the model on a separate gaming notebook, with nvidia, while playing the game on the one I'm using now.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#40

Post by spaceywilly »

hathawsh wrote: post #12

I would like to extend the question: is anyone building a homelab for the specific purpose of training a LLM on their personal info? The choice of hardware (for speed, cost, and noise concerns) seems important.

I think a good use case for this would be auto generation of code documentation. There are many reasons to not want to upload your source code to a cloud AI service, but having an AI that was trained on your local code base so you could ask it “what does this foobar function do anyways?” would be killer.
Post reply on HN