Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is anyone using self hosted LLM day to day and training it like a new employee

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#61

Post by abdullin »

valine wrote: post #7

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time. If your goal is to bake new concepts into the model weights, your only real option is a dataset with tha…

Embeddings can be tricky. They are just an average semantic vector over a chunk of text.

There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#62

Post by abdullin »

willsmith72 wrote: post #5

Everyone I know just uses the hosted ones, because of the sheer performance gap. For now, you can do all the custom/manual training you want, but gpt4 will almost always outperform it with the right context. Hopefully that will change in the future. Even then, I don't expect people to want to self-host as in on their own machines. More like custom training, then host either on SAAS or PAAS, or their own on-prem if th…

It doesn't need to be an exclusive choice. Hosted and local models can complement each other.

On one of the projects I used Chat GPT-4 to write instructions/tutorials that were then executed by local model on a large chunk of data (cleaning up product catalogues).

GPT-4 then reviewed some results and fine-tuned the instructions.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#63

Post by willsmith72 »

abdullin wrote: post #62
willsmith72 wrote: post #5

Everyone I know just uses the hosted ones, because of the sheer performance gap. For now, you can do all the custom/manual training you want, but gpt4 will almost always outperform it with the right context. Hopefully that will change in the future. Even then, I don't expect people to want to self-host as in on their own machines. More like custom training, then host either on SAAS or PAAS, or their own on-prem if th…

It doesn't need to be an exclusive choice. Hosted and local models can complement each other. On one of the projects I used Chat GPT-4 to write instructions/tutorials that were then executed by local model on a large chunk of data (cleaning up product catalogues). GPT-4 then reviewed some results and fine-tuned the instructions.

interesting, what does the local model do?

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#64

Post by valine »

abdullin wrote: post #61
valine wrote: post #7

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time. If your goal is to bake new concepts into the model weights, your only real option is a dataset with tha…

Embeddings can be tricky. They are just an average semantic vector over a chunk of text. There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.

It’s not perfect, if you know of a better alternative I would genuinely love to hear about it.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#65

Post by ljm »

jacquesm wrote: post #36
ijwann wrote: post #27

Earlier quoted context omitted.

> Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it. Seems very realistic!

No, it would be realistic if after two weeks on the job they start telling you how to run the company.

It should already have a published series of shelf-help books by that point

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#66

Post by abdullin »

willsmith72 wrote: post #63
abdullin wrote: post #62

Earlier quoted context omitted.

It doesn't need to be an exclusive choice. Hosted and local models can complement each other. On one of the projects I used Chat GPT-4 to write instructions/tutorials that were then executed by local model on a large chunk of data (cleaning up product catalogues). GPT-4 then reviewed some results and fine-tuned the instructions.

interesting, what does the local model do?

One example - a local model (Mistral 7B OpenChat 3.5, which ranks high in my benchmarks) was generating additional search keywords for products at online marketplace. A second run of the model was cleaning up bad keywords.

The fun part here - ChatGPT-4 reviewed some user searches, product details and comments of marketing department. Then it generated condensed tutorials on writing good keywords for the products in this system (keywords have to cover unexpected search terms that people would use when searching for a product).

The tutorial was supposed to be for the "junior marketing assistant", but in reality it was fed to Mistral 7B.

The second pass was done similarly. "Hey, ChatGPT, these are some sample keywords that are produced by junior marketing assistant according to your tutorial. Review them and write a short guide on correcting most common problems".

It works nicely.

Other cases of local models that work good - custom embeddings (multi-lingual, mapped to the same vector space), custom TTS, custom STT. These are mostly used for specialised personal assistants.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#67

Post by abdullin »

valine wrote: post #64
abdullin wrote: post #61

Earlier quoted context omitted.

Embeddings can be tricky. They are just an average semantic vector over a chunk of text. There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.

It’s not perfect, if you know of a better alternative I would genuinely love to hear about it.

If I absolutely need to avoid hallucinations (e.g. when building marketing/sales assistants for the businesses), then I allow LLM to control and drive search for the relevant documents.

On a high level:

(1) give LLM ability and enough information to "expand" user query into a multiple search phrases. Search engine will use them to find most relevant fragments via a form of embedding search

(2) Get highest ranking document fragments and "show" them to LLM saying: "This are the results that were found in the document database using your search phrases via embedding similarity. Refine the search"

(3) Repeat that a couple of times, then rank final documents and combine them for the final answer synthesis.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#68

Post by mtlynch »

LegitShady wrote: post #35
mtlynch wrote: post #22

Earlier quoted context omitted.

Why not do it the old fashioned way and hire a human for this? Humans also have the advantage that they don't just make up answers when they don't know something (or at least if you hire good ones). I've had good experience hiring support folks and working with them on a shared inbox (we use HelpScout).

he is the human that does this.

Right, but developers are significantly more expensive than customer support agents. Especially if the answers are already documented, and the rep just needs to be friendly in pointing customers to the right documentation.

If you can free up 5 hours of week of dev time by spending $100/week on a support rep, that's a great rate for dev time.

This assumes you're running a profitable business in the first place. If you're still struggling to find paying customers, then yeah it makes sense to do everything yourself.

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#69

Post by Havoc »

abdullin wrote: post #59

I've been building workflow assistants that make existing employees more productive or enable entirely new business models. Some of these assistants use selected local models (due to cost or privacy factors) Currently the stack is gravitates around: - GPT-4 - either to drive the entire workflow OR generate prompts, plans and guidelines for the local models to execute. - structured knowledge bases (either derived from…

Do you use a framework to pull it all together? Like Langchain etc

Re: Is anyone using self hosted LLM day to day and training it like a new employee

#70

Post by abdullin »

Havoc wrote: post #69
abdullin wrote: post #59

I've been building workflow assistants that make existing employees more productive or enable entirely new business models. Some of these assistants use selected local models (due to cost or privacy factors) Currently the stack is gravitates around: - GPT-4 - either to drive the entire workflow OR generate prompts, plans and guidelines for the local models to execute. - structured knowledge bases (either derived from…

Do you use a framework to pull it all together? Like Langchain etc

LangChain is good for the demos and learning, but it is too complex and brittle for my taste.

Using a bit of boilerplate code (a couple of python files) that I copy to new projects.

Post reply on HN