Ask HN: People who switched from GPT to their own models. How was it? • Hacker News Board

Re: Ask HN: People who switched from GPT to their own models. How was it?

#21

Post by osigurdson » Tue, Feb 27, 2024, 5:00 AM UTC

MaxLeiter wrote: post #15

My 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality

I greatly prefer to use ChatGPT-4 instead of 3.5 despite the slowness. Really a good feature for them to have would be to easily re-run a prompt on 4. However, the glitchiness of the service is kind of annoying.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#22

Post by [deleted] » Tue, Feb 27, 2024, 5:06 AM UTC

geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

[deleted]

Re: Ask HN: People who switched from GPT to their own models. How was it?

#23

Post by fbdab103 » Tue, Feb 27, 2024, 5:22 AM UTC

tuanm wrote: post #11

jianfgo wrote: post #8

Anyone has a tutorial how to achieve it to own a self-hosted model?

You can try going get some pre-trained (sometimes, fine-tuned) models on HuggingFace, following their instructions. Good luck!

Bit "Draw the rest of the owl" there.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#24

Post by fbdab103 » Tue, Feb 27, 2024, 5:25 AM UTC

jianfgo wrote: post #8

Anyone has a tutorial how to achieve it to own a self-hosted model?

I think the llm utility[0] (the one from Simon, not Google) is probably the best quickstart experience you can find. Gives the option to connect to services via API or install/run local models.

As simple as

  pip install llm
  # add the local plugin
  llm install llm-gpt4all
  # Download and run a prompt against the Orca Mini 7B model
  llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'

Alternatively, you could use the llamafile[1] which is a tiny binary runner which gets packaged ontop of the multigigabyte models. Download the llamafile and you can launch it through your terminal or a web browser.

From the llamafile page, after you download the file, you can just launch it as

  ./mistral-7b-instruct-v0.2.Q5_K_M.llamafile -ngl 9999 --temp 0.7 -p '[INST]Write a story about llamas[/INST]'

[0] https://llm.datasette.io/en/stable/index.html

[1] https://github.com/Mozilla-Ocho/llamafile

Edit: added llm quickstart from the intro page

Re: Ask HN: People who switched from GPT to their own models. How was it?

#25

Post by theolivenbaum » Tue, Feb 27, 2024, 5:26 AM UTC

We support both in our app and enterprise product. The APIs (OpenAI) vs libraries (i.e. llama.cpp for on-device) are so similar that the switch is basically transparent to the user. We're adding support for other platforms APIs soon, and everything we've looked so far is as easy to integrate as OpenAI - except Google that for some reason complicates everything on Google Cloud.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#26

Post by itake » Tue, Feb 27, 2024, 5:27 AM UTC

We have a first pass with our own model and then escalate to gpt if we aren't sure of our own model's results.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#27

Post by arbitrandomuser » Tue, Feb 27, 2024, 5:30 AM UTC

simonw wrote: post #19

m3kw9 wrote: post #13

Earlier quoted context omitted.

For narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.

Have you done this? How did you do it? I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!

this is imo the secret sauce that gives people an edge and not a lot of people will want to reveal

Re: Ask HN: People who switched from GPT to their own models. How was it?

#28

Post by kgeist » Tue, Feb 27, 2024, 5:33 AM UTC

The main hurdle is lack of multilingual support in open weights models.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#29

Post by armini » Tue, Feb 27, 2024, 5:34 AM UTC

Would be great if people could share their app demo, host & model used/trained for better context.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#30

Post by serf » Tue, Feb 27, 2024, 5:37 AM UTC

fbdab103 wrote: post #23

tuanm wrote: post #11

Earlier quoted context omitted.

You can try going get some pre-trained (sometimes, fine-tuned) models on HuggingFace, following their instructions. Good luck!

Bit "Draw the rest of the owl" there.

it's all pretty well put together nowadays honestly.

here's a dead simple way : (1) download LM Studio, install it[0] (2) download a model from within the client when prompted (3) have a ball.

the program is fairly intuitive, it takes care of finding the relevant files, and it can even accept addendum prompts and various ways to flavor or specialize answers.

Learn the basics there, take what you learn to a more 'industrial' playground later on.

[0]: https://lmstudio.ai/