Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: People who switched from GPT to their own models. How was it?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Ask HN: People who switched from GPT to their own models. How was it?

#61

Post by mdekkers »

weinzierl wrote: post #47

A couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way. The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?" To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the prese…

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

Re: Ask HN: People who switched from GPT to their own models. How was it?

#62

Post by iAkashPaul »

namanyayg wrote: post #16
iAkashPaul wrote: post #5

Running Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.

Hey Akash would you mind elaborating about what you mean by contract mining?

Basically augmenting users parsing PDFs & looking to prefill values into Excel instead of typing it all out. Ex. Liabilities, time-period/frequencies mentioned, owners of clauses etc.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#63

Post by 3abiton »

0x008 wrote: post #53
manca wrote: post #31

Earlier quoted context omitted.

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.

Same, CodeLlama 70B is known to suck. Deepseek is the best for coding so far in my experience, Mixtral 8x7B is another great contender (to be frank, for most tasks). Miqu is making a buzz, but so far I haven't tested it personally yet.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#64

Post by WhrRTheBaboons »

mdekkers wrote: post #61
weinzierl wrote: post #47

A couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way. The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?" To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the prese…

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

> where you cannot predict the outcome with a very high level of confidence

so never demo LLMs. got it.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#65

Post by waynesonfire »

WhrRTheBaboons wrote: post #64
mdekkers wrote: post #61

Earlier quoted context omitted.

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

> where you cannot predict the outcome with a very high level of confidence so never demo LLMs. got it.

Correct.. like how google does it

https://www.tomshardware.com/news/google-gemini-ai-video-sta...

Re: Ask HN: People who switched from GPT to their own models. How was it?

#66

Post by zingelshuher »

MaxLeiter wrote: post #15

My 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality

When using it for coding I'd rather have quality than garbage 3x faster and cheaper. So, I'm not moving. For narrow tasks it makes sense. Also interesting is the idea of having several models working together locally. With, may be, occasional ping to GPT-4.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#67

Post by ranguna »

codybontecou wrote: post #18

I've been using [continue]( https://continue.dev/ ) alongside Ollama. My go-to llm has been [deepseek-coder 7b]( https://ollama.com/library/deepseek-coder ). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

Have you tried dolphin mistral 7B 2.6?

Part of its training data was code.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#68

Post by alecco »

iAkashPaul wrote: post #5

Running Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.

> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available

Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK).

Some people report throttling issues in some cases, though.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#69

Post by quickgist »

exploringBytes wrote: post #60
visarga wrote: post #36

Earlier quoted context omitted.

> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones

More complex instructions should be split into multiple prompts, solving the challenge step by step

Done this way, you also re-incur any input token costs for every additional step. If you're using a large context, this can be significant.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#70

Post by [deleted] »

mdekkers wrote: post #61
weinzierl wrote: post #47

A couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way. The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?" To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the prese…

Never demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence

[deleted]
Post reply on HN