Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: People who switched from GPT to their own models. How was it?

news.ycombinator.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: Ask HN: People who switched from GPT to their own models. How was it?

#81

Post by gpjt »

0x008 wrote: post #53
manca wrote: post #31

Earlier quoted context omitted.

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.

What quantization were you using? I've been getting some weird results with 34b quantized to 4 bits -- glitching, dropped tokens, generating Java rather than Python as requested. But 7b, even at 4 bits, works OK. Posted about it earlier on this evening: https://www.gilesthomas.com/2024/02/llm-quantisation-weirdne...

Re: Ask HN: People who switched from GPT to their own models. How was it?

#82

Post by seeknotfind »

mdekkers wrote: post #61
weinzierl wrote: post #47

A couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way. The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?" To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the prese…

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

Why not? This seems like a great thing to happen. If the author's heart is really in the project, then this is a huge opportunity moment to learn something new.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#83

Post by hnfong »

WhrRTheBaboons wrote: post #64
mdekkers wrote: post #61

Earlier quoted context omitted.

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

> where you cannot predict the outcome with a very high level of confidence so never demo LLMs. got it.

If you tune down the temperature you can get a high level of confidence that the output will be the same.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#84

Post by samus »

0x008 wrote: post #54

Mixed results. I think llama2 in general is pretty bad, especially at anything else than english. I've had very good results with Mixtral for Chat. Of course all of them feel like a Frankenstein compared to actual ChatGPT. They feel similar and work just as well until, sometimes, they put out complete and utter garbage or artifacts and you wonder if they skimped on fine-tuning.

In a certain sense, all open models skimp on fine-tuning. Since ChatGPT gets user feedback, OpenAI theoretically sits on a growing pile of data that can be used for continuous fine-tuning and alignment. For local models, you'd have to look for new checkpoints every few months or track good and bad responses and do your own alignment.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#85

Post by apetresc »

ninetyninenine wrote: post #77
apetresc wrote: post #75

Earlier quoted context omitted.

> This also helps distributes traffic as a side effect. I'm not sure it's "distributing" traffic so much as amplifying it.

Two users. One user is talking about physics, the other about art. Two different models are utilized. Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.

Ah okay, I misunderstood. I thought you meant inferencing on all the different models and letting them judge to what extent they're the best fit.

What you're suggesting is just simply intent classification and using a specific model per intent. That's what everyone did _before_ LLMs.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#86

Post by geoduck14 »

maxlamb wrote: post #4
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

Which model did you start with?

Llama2 Code

Re: Ask HN: People who switched from GPT to their own models. How was it?

#87

Post by ninetyninenine »

apetresc wrote: post #85
ninetyninenine wrote: post #77

Earlier quoted context omitted.

Two users. One user is talking about physics, the other about art. Two different models are utilized. Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.

Ah okay, I misunderstood. I thought you meant inferencing on all the different models and letting them judge to what extent they're the best fit. What you're suggesting is just simply intent classification and using a specific model per intent. That's what everyone did _before_ LLMs.

Given the existence of specialized LLMs people can still do this.
Post reply on HN