Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

Post by [deleted] » Wed, Nov 01, 2023, 11:44 PM UTC

geedzmo wrote: post #308

alex-moon wrote: post #289

I tried my standard "trick" question I use for LLMs: "Give me five papers with code demonstrating the state of the art of machine learning which uses geospatial data (e.g. GeoJSON) as both input and output." There is no such state of the art. My hand-wavey understanding is that GIS data is non-continuous, which makes it useless for transformers, and also contextual, which makes it useless for anything else. Will defe…

I don't see how this would be relevant for a code model? The code model isn't trained to retrieve papers/articles, it's meant to complete code. Whether or not you find hallucination in a unrelated task isn't particularly interesting.

Post by alex-moon » Thu, Nov 02, 2023, 1:42 AM UTC

geedzmo wrote: post #308

alex-moon wrote: post #289

I tried my standard "trick" question I use for LLMs: "Give me five papers with code demonstrating the state of the art of machine learning which uses geospatial data (e.g. GeoJSON) as both input and output." There is no such state of the art. My hand-wavey understanding is that GIS data is non-continuous, which makes it useless for transformers, and also contextual, which makes it useless for anything else. Will defe…

I don't see how this would be relevant for a code model? The code model isn't trained to retrieve papers/articles, it's meant to complete code. Whether or not you find hallucination in a unrelated task isn't particularly interesting.

Post by riku_iki » Thu, Nov 02, 2023, 2:23 PM UTC

emptysongglass wrote: post #344

riku_iki wrote: post #331

Earlier quoted context omitted.

lol, you can continue trusting anecdotes from internet. Industry prefers more scientific methods.

So Paul Graham posted that Phind is better and got absolutely destroyed in the comments https://twitter.com/paulg/status/1719657855240815026 No, I do not take these benchmarks seriously and for good reason. They're benchmarks. The only thing that matters is the user's direct experience of the product. And Phind isn't there.

Post by blago » Thu, Nov 02, 2023, 6:15 PM UTC

pbhjpbhj wrote: post #150

blago wrote: post #138

It failed for me at a much more basic level. I asked 5 different, and increasing explicit, variations of the following question: "Can you generate HTML and CSS for a JPG mockup I'm going to give you?" Each time it answered along the following lines: "Sure, here is how you can create HTML and CSS from a JPG mockup. Follow this process..." In my experience this never happens with GPT-4.

I've not seen that anywhere, ChatGPT does image input now? Do you have examples of the output from feeding it a JPEG?

Post by a_wild_dandan » Fri, Nov 03, 2023, 12:52 AM UTC

Der_Einzige wrote: post #228

a_wild_dandan wrote: post #155

Earlier quoted context omitted.

They are. Moreover, the idea that AI companies are missing and/or not implementing this “obvious” tactic is hilarious. Folks, these approaches have profound consequences for training and inference performance. Y’all aren’t pointing out some low hanging fruit here, lol

Actually, yes I am pointing out low hanging fruit here. These approaches do not have "profound consequences" for inference or training performance. In fact, sentence transformer models run orders of magnitude more quickly. Performance penalties will be small. Also, I actually have several top NLP conference publications, so I'm not some charlatan when I say these things. I've actually physically used and seen these t…