Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is anyone using self hosted LLM day to day and training it like a new employee

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#11

Post by abstrct »

bentt wrote: post #9
abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.

I’ve absolutely explored this idea but, similar to lossy compression, sometimes important nuance is lost in the process. There is both an art and science to recalling the gently compacted information and being able to recognize when it needs to be repeated back.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#13

Post by abstrct »

coryrc wrote: post #10
abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

I think you'll need to save good responses (and bad responses that you fixed?) and regularly run more training passes.

Yeah, especially with a large knowledge base I find it important to keep a log of prompts/responses and perform team reviews of both. It’s honestly making more work than it’s saving at the moment with the hope that it’ll be more helpful down the road. On the plus side it’s made the team more interested in tasks around technical documentation and marketing material, so still a win!

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#14

Post by filterfiber »

bentt wrote: post #9
abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.

Yeah that's a fairly well studied one. Most of these techniques are rather "lossy" compared to extending the context window. The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window.

Here's a bunch of other related methods,

Summarizing context - https://arxiv.org/abs/2305.14239

continuous finetuning - https://arxiv.org/pdf/2307.02839.pdf

retrieval augmented generation - https://arxiv.org/abs/2005.11401

knowledge graphs - https://arxiv.org/abs/2306.08302

augmenting the network a side network - https://arxiv.org/abs/2306.07174

another long term memory technique - https://arxiv.org/abs/2307.02738

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#15

Post by kolinko »

TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

I’m working on this right now, will be open sourcing soon probably :)

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#16

Post by willsmith72 »

TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

I've been thinking about the 'glue' bit too.

I think a cron job run say every hour would be good enough. It would just have to collect all the inputs and make a decision about how "valuable" it would be to check in (haven't worked out in 3 days, very valuable), then it's just about connecting it to the right outputs (email, push noti, ...)

The job itself would be cheap and trivial to host on something like lambda

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#17

Post by sureglymop »

TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

Just be aware of the security implications. Maybe it's unrealistic but what if someone sends you a calendar invite containing a prompt injection? It may not seem like a big issue but at worst (e.g. with github copilot) something like this may lead to remote code execution.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#18

Post by Casteil »

TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

Having come from Notion also, I LOVE Obsidian for its non-proprietary markdown file structure. Incredible powerful plugins, too.

FWIW, it's not really what you're seeking... but there is a plugin that allows you to invoke an LLM from within Obsidian (via Ollama): https://github.com/hinterdupfinger/obsidian-ollama

In short, it allows you to set up prompts to act on selected text directly within a file, e.g. 'Summarize this selection as a markdown formatted list of key points', 'Write a PRD', 'Translate to [Language]', 'Run this as a prompt', etc.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#19

Post by adammichaelc »

abstrct wrote: post #11
bentt wrote: post #9

Earlier quoted context omitted.

Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.

I’ve absolutely explored this idea but, similar to lossy compression, sometimes important nuance is lost in the process. There is both an art and science to recalling the gently compacted information and being able to recognize when it needs to be repeated back.

If there was something like Objects in OO programming, but for LLM’s, would that solve this?

Like a Topic-based Personality Construct where the model first determines which of its “selves” should answer the question, and then grabs appropriate context given the situation.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#20

Post by htrp »

semireg wrote: post #8

As a solo developer answering emails that basically point people to various guides and FAQs I’ve published … I need this. Zendesk claims to have an AI component but forces you to input all training data into their own wiki knowledge base. I can see why they don’t want to use prior responses as training (pii concerns), but at least give me some boilerplate responses that I can use to get a head start and further train…

You just need embedding based search.
Post reply on HN