A few of Phind's models are open/available
Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
221–230 of 358 posts
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#222Could you open source these great models? OK yes you need a competitive advantage. So maybe open source them when you are say 2 models ahead in production? In any case I am happy there is some competition and that it has come from a more pragmatic scrappy space than one of the multiple billion dollar funded places.
I don't know what model runs on Phind's site right now, but in August Phind published a fine tune of CodeLlama 34B
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#223Earlier quoted context omitted.
> * Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research. Did you find them to be correct?
I don't think "correct" is the right word since these were open ended systems design type questions. There are many ways to accomplish the same task. I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it. In my experience, the sources were sufficiently relevant based on its responses. They were a…
And the sources actually existed? i.e. there weren't any made-up ones?
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#224Earlier quoted context omitted.
that would put them significantly ahead again, for my use cases
We will eventually increase the Phind Model to 100K tokens -- the RoPE embeddings in Code Llama were designed for this.
> the RoPE embeddings in Code Llama were designed for this.
The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE (e.g. position interpolation) are post-hoc tweaks (with optional tuning) to an entirely vanilla RoPE implementation.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#225I just spent a few minutes doing a comparison between Phind and GPT-4 for a very high-level question on a distributed job queue. I gave them both the same fairly vague sketch of a kind of system I would like to build. Here are my impressions: In the positives of Phind: * Phind was able, even eager, to recommend specific libraries relevant to the implementation. The recommendations matched my own research. GPT-4 takes…
The “give context” part has a lot to do with prompting well based on the model. To have a fair comparison there should be just code and see what that come up with
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#226Earlier quoted context omitted.
that would put them significantly ahead again, for my use cases
We will eventually increase the Phind Model to 100K tokens -- the RoPE embeddings in Code Llama were designed for this.
Is it “100k” or really 100k there are so many ways to do context, I remember seeing 100k before but it was doing some cheap trick to get it
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#227Earlier quoted context omitted.
I don't think "correct" is the right word since these were open ended systems design type questions. There are many ways to accomplish the same task. I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it. In my experience, the sources were sufficiently relevant based on its responses. They were a…
And the sources actually existed? i.e. there weren't any made-up ones?
The sources are urls to the cited page (e.g. stackoverflow.com, pkg.go.dev). In the side-bar next to the answer is a more standard search-result style link list with pulled quotes from the pages (like a Google search).
I didn't click every single link (as I mentioned, the citations are copious) but the few I did follow went to relevant articles. I just went back and randomly clicked several more and they all went to pages that exist and mostly relate to the content of the answer. The inline citations seem a bit more on-topic compared to the side bar which does seem more like the links were lifted directly from a search engine.
To be fair there are some lower-quality blog-spammy kinda stuff - more or less the same kind of thing you would get out of Google. But compared to GPT-4, which provides no sources whatsoever, it is an advantage IMO.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#228Earlier quoted context omitted.
But some of the earlier tokens are also the most important ones, right? Like the instructions and rules you want it to follow.
They are. Moreover, the idea that AI companies are missing and/or not implementing this “obvious” tactic is hilarious. Folks, these approaches have profound consequences for training and inference performance. Y’all aren’t pointing out some low hanging fruit here, lol
Actually, yes I am pointing out low hanging fruit here. These approaches do not have "profound consequences" for inference or training performance. In fact, sentence transformer models run orders of magnitude more quickly. Performance penalties will be small.
Also, I actually have several top NLP conference publications, so I'm not some charlatan when I say these things. I've actually physically used and seen these techniques improve LLM recall. It really actually works.
Here's more examples of low hanging fruit. The proof in that they work is in the implementations which I provide. You can run them, they work!: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
Check yourself before you try to check others.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#229Earlier quoted context omitted.
Still waiting for the day that medium term memory (token average pooling like in sentence transformers) becomes used for this. It's staring all of these companies in the face and apparently no one thinks to implement it.
Out of curiosity, why do you think the answer would be so simple and also completely untested?
Too much money being thrown around on BS in the LLM space, hardly any of it is going to places where it matters. Ignorance on the part of investors.
For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring in relative obscurity.
HN, and the NLP community have some serious blindspots with knowing how to exploit their own technology. At least someone at Andreessen Horowitz got a clue and gave some funding to Oogabooga - still waiting for Automatic1111 to get any funding.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#230I am a heavy user of GPT4, and Phind was surprisingly able to match GPT4 on several initial programming tasks I gave it. Given the large context window of Phind, it will likely be able to outperform GPT4 for some tasks. That is quite an accomplishment, I am impressed
FWIW The default context window of GPT-4 via ChatGPT is about to change to 32k.
This would be great if true. Any source for this?