Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is anyone using self hosted LLM day to day and training it like a new employee

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#51

Post by benterix »

filterfiber wrote: post #14
bentt wrote: post #9

Earlier quoted context omitted.

Are there methods to "summarize what they've learned" and then replace the context window with the shorter version? This seems like pretty much what we do as humans anyway... we need to encode our experiences into stories to make any sense of them. A story is a compression and symbolization of the raw data one experiences.

Yeah that's a fairly well studied one. Most of these techniques are rather "lossy" compared to extending the context window. The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. Here's a bunch of other related methods, Summarizing context - https://arxiv.org/abs/2305.14239 continuous finetuning - https://arxiv.org/pdf/2307.0283…

Is there a realistic way to actually increase the context window?

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#52

Post by pcdoodle »

TrevorJ wrote: post #2

I'm pretty interested in this as well. I have moved from Notion to Obsidian for my personal notes, to-do lists and errata in preparation for this since obsidian uses local plaintext files. What I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today th…

I like it.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#53

Post by filterfiber »

benterix wrote: post #51
filterfiber wrote: post #14

Earlier quoted context omitted.

Yeah that's a fairly well studied one. Most of these techniques are rather "lossy" compared to extending the context window. The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. Here's a bunch of other related methods, Summarizing context - https://arxiv.org/abs/2305.14239 continuous finetuning - https://arxiv.org/pdf/2307.0283…

Is there a realistic way to actually increase the context window?

Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models.

However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC.

Here's a decent overview https://aman.ai/primers/ai/context-length-extension/

Rope - https://arxiv.org/abs/2306.15595

Yarn (based on rope) - https://arxiv.org/pdf/2309.00071.pdf

LongLoRA - https://arxiv.org/pdf/2309.12307.pdf

The bottleneck is quickly going to be inference. Since the current transformer models need the context length ^2, the memory requirements go up very quickly. IIRC a 4090 can _barely_ fit a 4bit 30B model in memory with 4096k context length.

From my understanding some form of RNNs are likely to be the next step for longer context. See RWKV as an example of a decent RNN https://arxiv.org/abs/2305.13048

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#54

Post by shanelleroman »

I've been using & contributing to Lightrail (https://github.com/lightrail-ai/lightrail). Each instance comes with a local vectorDB and integrates with apps like Chrome & VSCode, so I can read in content like my notes, emails, etc. It doesn't support self-hosted LLMs yet unfortunately!

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#55

Post by Belomolo »

abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

Funny but also true in real life :-(

I start to feel like a one eye king under blind people.

I even remember sometimes when I told people specific things.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#56

Post by nicbou »

mtlynch wrote: post #22
semireg wrote: post #8

As a solo developer answering emails that basically point people to various guides and FAQs I’ve published … I need this. Zendesk claims to have an AI component but forces you to input all training data into their own wiki knowledge base. I can see why they don’t want to use prior responses as training (pii concerns), but at least give me some boilerplate responses that I can use to get a head start and further train…

Why not do it the old fashioned way and hire a human for this? Humans also have the advantage that they don't just make up answers when they don't know something (or at least if you hire good ones). I've had good experience hiring support folks and working with them on a shared inbox (we use HelpScout).

I considered both options for allaboutberlin.com. I want to offer either better search, or personalised advice.

Cost is a huge problem. An immigration lawyer would continuously eat into my personal income. If I don't get state funding to cover it, it makes zero business sense. It would also come with all the liability of a first employee.

A GPT that cites my website, German law and a dozen official websites would be a game changer. It could not give advice, but it could find answers like Phind does. It's just tech, with lower running costs and virtually no obligations (unlike an employee).

I just don't know if the result would be useful and trustworthy enough, and it's very expensive to try.

My conclusion is that I should focus on building a good knowledge base, and when the time is ripe, I can augment it with fancy tech.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#57

Post by dmezzetti »

Cool use case, glad to see txtai [1] is helping (I'm the main dev for txtai).

Since you're using txtai, this article I just wrote yesterday might be helpful: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai

Looks like you've received a lot of great ideas here already though!

1 - https://github.com/neuml/txtai

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#58

Post by johntash »

I have a similar goal/desire as you. My current project is ingesting this type of data into Elasticsearch with vector embeddings and using a normal search+knn to generate some context when creating a prompt.

This works reasonably well with gpt4, but my context is almost always too large for self-hosted models so far.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#59

Post by abdullin »

I've been building workflow assistants that make existing employees more productive or enable entirely new business models. Some of these assistants use selected local models (due to cost or privacy factors)

Currently the stack is gravitates around:

- GPT-4 - either to drive the entire workflow OR generate prompts, plans and guidelines for the local models to execute.

- structured knowledge bases (either derived from existing sources OR curated manually by companies to drive AI assistants).

- embedding search indexes, augmented by full-text search. Usually LLM has access to the search engine and can drive the search as needed, refining the queries if results aren't good enough.

All of that is instrumented with logic to capture user feedback at every single step. This is crucial for the continuous improvement of the model!

Bigger model can use this information once in a while, to improve plans and workflow guidelines to make the overall process more efficient.

AMA, if needed!

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#60

Post by abdullin »

abstrct wrote: post #4

The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.

Putting too much information in the context window is counter-productive in my experience. Low signal/noise ratio tends to increate the likelihood of model hallucinations, and we don't want that!

What works in my experience - structuring the task similar to a human-driven workflow, breaking it down into small steps is needed. Each step could be driven by a small prompt, relevant document fragments (if RAG is used) and condensed essays/tutorials/guides that were written by a powerful LLM (ideally, GPT-4 pre-Turbo).

Using this approach, you could stay well below 8k token limit even on the most demanding tasks.

(Big size contexts are leaky on all LLMs anyway)

Post reply on HN