241–250 of 358 posts
I was just testing to see the comparison and ran into a message saying I was out of GPT-4 queries, despite having deliberately selected Phind as my model. Now I'm confused if the results I was seeing really were from a different model than GPT-4 or not.
Earlier quoted context omitted.
I think you can bring contamination claim to every public benchmark results nowdays: models are trained on TBs of data crawled from internet, and there is no guarantee benchmark is not leaked in some way.
With respect to the pretraining data, its true that we're probably SOL there in terms of verification. But for fine-tuning, they could still publish the dataset and see if others can reproduce their results as well as audit for contamination. If we're comparing benchmark deltas between different fine-tuned variants that share the same base models, that seems like the bare minimum we should expect to come along with p…
I just spent a few minutes doing a comparison between Phind and GPT-4 for a very high-level question on a distributed job queue. I gave them both the same fairly vague sketch of a kind of system I would like to build. Here are my impressions: In the positives of Phind: * Phind was able, even eager, to recommend specific libraries relevant to the implementation. The recommendations matched my own research. GPT-4 takes…
Edit: they are already posted as comment in this thread.
I was just testing to see the comparison and ran into a message saying I was out of GPT-4 queries, despite having deliberately selected Phind as my model. Now I'm confused if the results I was seeing really were from a different model than GPT-4 or not.
Ah you were likely using the Pair Programmer. The Phind Model is not yet supported in the Pair Programmer, only the default search mode. Please try again using that.
Earlier quoted context omitted.
LOL, it’s not just for “public acceptance”. Look up Chain of Thought. Asking it to get to the point typically reduces the accuracy.
You are being snarky but are right. I have scripts set up to auto summarise expansive answers. I wish I could build this into the ChatGPT ui though.
I tried this, but I still have yet to get any LLM to answer me a programming question (that actually works) that I actually want to solve. Basically: "How can I send network control commands to an AppleTV in C#" They always make up some nonexistent library or gives an example using some nonexistent API.
Interesting, I seem to have gotten a decent answer: https://www.phind.com/search?cache=avbridtm69ejk8pdqpx8hcnf
I love that Phind cites what it scrapes. This should be the obligation of all LLM. I always suggest people use it over ChatGPT.
As a user, i perfer getting the right response compared to the thing spitting out a link. (not saying phind is bad). Lets focus on getting llm right before nerfing it in its baby stages.
Earlier quoted context omitted.
With respect to the pretraining data, its true that we're probably SOL there in terms of verification. But for fine-tuning, they could still publish the dataset and see if others can reproduce their results as well as audit for contamination. If we're comparing benchmark deltas between different fine-tuned variants that share the same base models, that seems like the bare minimum we should expect to come along with p…
I think both pretraining and finetuning datas are essential secret information for commercial models/services.
Even I am to grant that their subscription product has some secret sauce they want to keep close to the chest (ignoring for a moment their paid product is GPT-4 based), not doing the same for all the models they release to the open source community free of charge with a commercially-permissible license seems suspect.
I realize this sort of open source contribution is mostly for marketing purposes, but being critical of the performance claims I think is still valid nonetheless.
Earlier quoted context omitted.
Interesting, I seem to have gotten a decent answer: https://www.phind.com/search?cache=avbridtm69ejk8pdqpx8hcnf
Unfortunately there’s nothing correct about that answer. There’s no tcp service listening for requests like that on port 7000 on an AppleTV.
Earlier quoted context omitted.
Who said anything about nerfing? Citation is just additive, no?
In fact, I’d argue that citation makes LLM better. Kind of a “think carefully” indicator. When LLMs are able to verify those citations independently it’s going to level up again by skyrocketing the objective truthiness.