But you would lose the benefit of self hosting
Is anyone using self hosted LLM day to day and training it like a new employee
21–30 of 75 posts
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#22As a solo developer answering emails that basically point people to various guides and FAQs I’ve published … I need this. Zendesk claims to have an AI component but forces you to input all training data into their own wiki knowledge base. I can see why they don’t want to use prior responses as training (pii concerns), but at least give me some boilerplate responses that I can use to get a head start and further train…
Why not do it the old fashioned way and hire a human for this? Humans also have the advantage that they don't just make up answers when they don't know something (or at least if you hire good ones).
I've had good experience hiring support folks and working with them on a shared inbox (we use HelpScout).
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#23Local models have taken a mind boggling leap over the past months so i'm sure we'll be able to add layers soon by ourselves even on a laptop?
Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air:
https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF
But yeah current context windows are limiting.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#24I would like to extend the question: is anyone building a homelab for the specific purpose of training a LLM on their personal info? The choice of hardware (for speed, cost, and noise concerns) seems important.
I admittedly have not done this myself, but the hardware choices are less complex than you might think. The only truly important part is the GPU, and the number of GPUs on the consumer market that can handle a local LLM is presently quite small. It's pretty much dual-wielding 3090's or 4090's or bust. If you're running but not training a (relatively) small one, you can do well with just one. But if you want to run inference _and_ training on one large enough to be a consistent daily assistant (read: 70b+), cloud hardware for the training is just far more practical and economical. You're generally not gonna wanna drop the $$ for an A100 or two just to do a small training run every couple of weeks on a dataset made up of the amount of RLHF'd conversations one person might have with an LLM in that time.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#25I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…
> What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today this might be a good time". That sort of thing.
If you work in a Microsoft world, this is what GraphAPI is all about: enabling access to all the things using your personal authentication token. This includes emails, calendar, OneNote, One Drive, and essentially everything. I've been working on making an easy to use OneNote provider with GraphAPI underneath it that I can use to work with the LLM.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#26I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…
Just be aware of the security implications. Maybe it's unrealistic but what if someone sends you a calendar invite containing a prompt injection? It may not seem like a big issue but at worst (e.g. with github copilot) something like this may lead to remote code execution.
Yeah, it's a good point. Have a feeling we will see a lot of sneaky security issues around LLM's over the next few years to say the least.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#27The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
> Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
Seems very realistic!
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#28I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…
Having come from Notion also, I LOVE Obsidian for its non-proprietary markdown file structure. Incredible powerful plugins, too. FWIW, it's not really what you're seeking... but there is a plugin that allows you to invoke an LLM from within Obsidian (via Ollama): https://github.com/hinterdupfinger/obsidian-ollama In short, it allows you to set up prompts to act on selected text directly within a file, e.g. 'Summarize…
That's a pretty slick plugin I will have to check it out, thanks for the suggestion.
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#29Earlier quoted context omitted.
Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.
Yeah that's a fairly well studied one. Most of these techniques are rather "lossy" compared to extending the context window. The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. Here's a bunch of other related methods, Summarizing context - https://arxiv.org/abs/2305.14239 continuous finetuning - https://arxiv.org/pdf/2307.0283…
this is a fantastically useful comment. thank you filterfiber :)
Re: Is anyone using self hosted LLM day to day and training it like a new employee
#30I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…
I’m working on this right now, will be open sourcing soon probably :)
That's pretty cool - let us know if you do!