Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: People who switched from GPT to their own models. How was it?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 87 posts

Re: Ask HN: People who switched from GPT to their own models. How was it?

#31

Post by manca »

codybontecou wrote: post #18

I've been using [continue]( https://continue.dev/ ) alongside Ollama. My go-to llm has been [deepseek-coder 7b]( https://ollama.com/library/deepseek-coder ). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#32

Post by ipaddr »

simonw wrote: post #19
m3kw9 wrote: post #13

Earlier quoted context omitted.

For narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.

Have you done this? How did you do it? I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!

https://www.datacamp.com/tutorial/fine-tuning-llama-2

Re: Ask HN: People who switched from GPT to their own models. How was it?

#33

Post by manca »

jianfgo wrote: post #8

Anyone has a tutorial how to achieve it to own a self-hosted model?

If you don't care about the details of how those model servers work, then something that abstracts out the whole process like LM Studio or Ollama is all you need.

However, if you want to get into the weeds of how this actually works, I recommend you look up model quantization and some libraries like ggml[1] that actually do that for you.

[1] https://github.com/ggerganov/ggml

Re: Ask HN: People who switched from GPT to their own models. How was it?

#34

Post by jwells89 »

codybontecou wrote: post #18

I've been using [continue]( https://continue.dev/ ) alongside Ollama. My go-to llm has been [deepseek-coder 7b]( https://ollama.com/library/deepseek-coder ). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

deepseek-coder 6.7b is seriously impressive for how quickly it runs on an M1 Max. There’s a few spots where it still doesn’t fare quite as well as ChatGPT but it’s a small tradeoff considering that it’s fully local and doesn’t even spin up my laptop’s fans.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#35

Post by anon373839 »

simonw wrote: post #19
m3kw9 wrote: post #13

Earlier quoted context omitted.

For narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.

Have you done this? How did you do it? I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!

This is a very nice resource: https://github.com/mlabonne/llm-course

Re: Ask HN: People who switched from GPT to their own models. How was it?

#36

Post by visarga »

geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

> GPT 4 isn't that great, in general

same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones

Re: Ask HN: People who switched from GPT to their own models. How was it?

#38

Post by arthur_sav »

MaxLeiter wrote: post #15

My 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality

OpenAI currently leads the front on (commercial) AI, so I doubt people will switch. In fact most offerings become outdated pretty fast and everyone else tries to play catch up.

Imagine using a GPT-2 type model when everyone else is using GPT-4. Until the dust settles there's no point in investing in alt models imo, unless you're leading the research.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#40

Post by wesleyyue »

I tested a bunch of models while building https://double.bot but ended up back on gpt4. Other models are fun to play with but it gets frustrating even if they miss 1/100 questions that gpt4 gets. I find that right now I get more value implementing features around the model that fixes all the GitHub copilot papercuts (autocomplete that closes brackets properly, auto import upon accepting suggestions, disable suggestions when writing comments to be less distracting, midline completions, etc etc)

Hopefully os models can catch-up to gpt4 in the next six months when we fixed all the low hanging fruit outside of the model itself

Post reply on HN