Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

The results I get are so-so. The rubric I use to evaluate coding LLM's is to ask it to create a Python script that determines if the contents of a given directory have been changed since the last time the script was run. This should be done recursively and handle files being added, removed, or modified and be based off the contents of the files and not the timestamps.

When I ask it as one statement it performed ok, but if I made more specifications with follow-up statements, it kept trying to go down one path even though I told it to do it a different way. A solid start but it definitely needs some improvements, IMO.

Fifth sentence:

> However, we’ve found that HumanEval is a poor indicator of real-world helpfulness.

It's so fast ... and accurate.

> We can achieve up to 100 tokens per second single-stream while GPT-4 runs around 20 tokens per second at best.

Is that with batching? If so, thats quite impressive.

> certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4.

Some of this is sampler tuning. Y'all should look at grammar based sampling (https://github.com/ggerganov/llama.cpp/pull/1773) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and dynatemp: https://github.com/LostRuins/koboldcpp/pull/464

I think these should work with nvidia's implementation if you just swap the sampling out with the HF version.

BTW, all this is a great advantage of pulling away from OpenAI. You can dig in and implement experimental features that you just can't necessarily do through their API.

Will you be offering the model as an API service? The product my team is working on would benefit from a significantly faster and possibly better performing model than GPT-4. If you're planning on keeping pace with competitive models we'd love to integrate the use of your model into our service.