Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is anyone using self hosted LLM day to day and training it like a new employee

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 75 posts

Is anyone using self hosted LLM day to day and training it like a new employee

#1

Post by reachableceo »

I have this idea to use LLM daily. Train it on my emails / notes / chats . Have it draft replies and I edit them as needed and it learns from that.

Is anyone doing anything like that? I have all of the open source stuff downloaded (models , lollms-webui , promptfoo, etc ) and have been experimenting with the interactive chat stuff . Also txtai to make semantic search.

That all seems pretty mature / progressing nicely . A few more months and I expect a clear reference stack will emerge .

What about the assistant stack ? I invest all these resources to self host and feed in all my data. I want to maximize the ROI.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#2

Post by TrevorJ »

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files.

What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today this might be a good time". That sort of thing.

There seem to be good tooling around agents, prompt engineering, RAG etc. The 'glue' around getting the LLM to help figure out when appropriate time(s) to check in with me is the bit I am missing, but that's probably mostly down to me being an artist and only a very very JR hobbyist programmer though.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#5

Post by willsmith72 »

Everyone I know just uses the hosted ones, because of the sheer performance gap.

For now, you can do all the custom/manual training you want, but gpt4 will almost always outperform it with the right context.

Hopefully that will change in the future. Even then, I don't expect people to want to self-host as in on their own machines. More like custom training, then host either on SAAS or PAAS, or their own on-prem if they have it. Spending the performance of a personal laptop isn't worth the reduction of performance on other tasks. Again, maybe that will change.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#6

Post by TheCaptain4815 »

This is something I imagine coming out of Autogen or OpenAi Assistants in a few months. You really need multiple agents (as of now) most of the time. IMO multiple GPT4 agents ARE smart enough to accomplish a lot, it's getting them working together and setup that's the issue.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#7

Post by valine »

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time.

If your goal is to bake new concepts into the model weights, your only real option is a dataset with that concept being used in a wide variety of contexts.

A more feasible approach I think would be retrial augmented generation. You’d essentially store your conversations in a database and calculate embeddings as you go. This would allow you to later do a natural language search of the database, and insert the most relevant portion of the conversation into your context window.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#8

Post by semireg »

As a solo developer answering emails that basically point people to various guides and FAQs I’ve published … I need this. Zendesk claims to have an AI component but forces you to input all training data into their own wiki knowledge base. I can see why they don’t want to use prior responses as training (pii concerns), but at least give me some boilerplate responses that I can use to get a head start and further train the model(s).

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#9

Post by bentt »

abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#10

Post by coryrc »

abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

I think you'll need to save good responses (and bad responses that you fixed?) and regularly run more training passes.
Post reply on HN