Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

1–10 of 358 posts

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#1

Post by rushingcreek »

Hi HN,

We’re excited to announce that Phind now defaults to our own model that matches and exceeds GPT-4’s coding abilities while running 5x faster. You can now get high quality answers for technical questions in 10 seconds instead of 50.

The current 7th-generation Phind Model is built on top of our open-source CodeLlama-34B fine-tunes that were the first models to beat GPT-4’s score on HumanEval and are still the best open source coding models overall by a wide margin: https://huggingface.co/spaces/bigcode/bigcode-models-leaderb....

This new model has been fine-tuned on an additional 70B+ tokens of high quality code and reasoning problems and exhibits a HumanEval score of 74.7%. However, we’ve found that HumanEval is a poor indicator of real-world helpfulness. After deploying previous iterations of the Phind Model on our service, we’ve collected detailed feedback and noticed that our model matches or exceeds GPT-4’s helpfulness most of the time on real-world questions. Many in our Discord community have begun using Phind exclusively with the Phind Model despite also having unlimited access to GPT-4.

One of the Phind Model’s key advantages is that it's very fast. We’ve been able to achieve a 5x speedup over GPT-4 by running our model on H100s using the new TensorRT-LLM library from NVIDIA. We can achieve up to 100 tokens per second single-stream while GPT-4 runs around 20 tokens per second at best.

Another key advantage of the Phind Model is context – it supports up to 16k tokens. We currently allow inputs of up to 12k tokens on the website and reserve the remaining 4k for web results.

There are still some rough edges with the Phind Model and we’ll continue improving it constantly. One area where it still suffers is consistency — on certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4.

We’d love to hear your feedback.

Cheers,

The Phind Team

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
phind.com

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#4

Post by rushingcreek »

pleonasticity wrote: post #3

This is great work, but HumanEval is an extremely limited benchmark and I don’t think you can seriously claim to beat GPT-4 at coding based only on that metric.

Thank you. You're right -- which is why we rely on feedback we've received from our own users for that claim. Many of our users who have the choice to use either GPT-4 or the Phind Model on Phind choose the Phind Model.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#6

Post by rushingcreek »

v3ss0n wrote: post #5

from my last week test of opensource model , it keep repeating and gives out broken outputs, using q4

This V7 model is much better than the V2 model that we previously open-sourced. And Q4 quantization would also likely have a large detrimental impact.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#7

Post by pleonasticity »

rushingcreek wrote: post #4
pleonasticity wrote: post #3

This is great work, but HumanEval is an extremely limited benchmark and I don’t think you can seriously claim to beat GPT-4 at coding based only on that metric.

Thank you. You're right -- which is why we rely on feedback we've received from our own users for that claim. Many of our users who have the choice to use either GPT-4 or the Phind Model on Phind choose the Phind Model.

I understand, but big claims require big evidence and so it’s still IMHO not rhetorically a strong position. I’m glad people find it more useful!

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#8

Post by Kranar »

rushingcreek wrote: post #4
pleonasticity wrote: post #3

This is great work, but HumanEval is an extremely limited benchmark and I don’t think you can seriously claim to beat GPT-4 at coding based only on that metric.

Thank you. You're right -- which is why we rely on feedback we've received from our own users for that claim. Many of our users who have the choice to use either GPT-4 or the Phind Model on Phind choose the Phind Model.

You likely know this, but keep in mind the kind of selection bias in taking feedback mostly from your own users. The number of times I've heard product designers claim that their users prefer some aspect of how their application already works, ignoring the fact that the users who didn't prefer it have left and hence are likely not available to survey.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#10

Post by rushingcreek »

Kranar wrote: post #8
rushingcreek wrote: post #4

Earlier quoted context omitted.

Thank you. You're right -- which is why we rely on feedback we've received from our own users for that claim. Many of our users who have the choice to use either GPT-4 or the Phind Model on Phind choose the Phind Model.

You likely know this, but keep in mind the kind of selection bias in taking feedback mostly from your own users. The number of times I've heard product designers claim that their users prefer some aspect of how their application already works, ignoring the fact that the users who didn't prefer it have left and hence are likely not available to survey.

Of course. We do our best to talk to churned users as well, but we're doing this Show HN to get even more diverse feedback.
Post reply on HN