Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

141–150 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#141

Post by ta8645 »

rushingcreek wrote: post #92
cccybernetic wrote: post #78

I'm glad there's growing competition, but GPT4 still outperforms. Here's a recent question I asked: Could you please provide me with a query for my PostgreSQL table? It consists of three columns: 'id,' 'teaser,' and 'full_text,' all of which are of type 'text.' Most 'full_text' entries are articles around 1000 words in length. I'd like to update the 'teaser' column for each row with the first approximately 200 words…

Running "Ignore Web Context" enabled can improve performance for design tasks like this. I just got a more plausible answer: https://www.phind.com/search?cache=f0fkv5mxscwvagxgkuwnwgtl . Consistency is something we're working on.

How do you enable "Ignore Web Context"? I don't see that option anywhere on the page you linked, am I just being blind?

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#143

Post by rushingcreek »

ta8645 wrote: post #141
rushingcreek wrote: post #92

Earlier quoted context omitted.

Running "Ignore Web Context" enabled can improve performance for design tasks like this. I just got a more plausible answer: https://www.phind.com/search?cache=f0fkv5mxscwvagxgkuwnwgtl . Consistency is something we're working on.

How do you enable "Ignore Web Context"? I don't see that option anywhere on the page you linked, am I just being blind?

It's in the model dropdown under the search bar.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#144

Post by rushingcreek »

ShakataGaNai wrote: post #139

"Python script to extract a list of all Elastic IP's from all regions, from multiple AWS accounts." ChatGPT4 gave me a solid answer hitting all the points I wanted. Phind din't get the account handling correct, didn't address regions, and didn't handle pagination. "Write a python based script that uses boto3 to query AWS Route53. It should print a list of every record for a given hosted zone ID." ChatGPT4 did exactly…

Just worked well for me: https://www.phind.com/search?cache=g9y2uizgjwcn378aovb65v92.

We do have issues with consistency sometimes -- please try regenerating if that is the case.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#145

Post by doctoboggan »

rushingcreek wrote: post #118
shazar wrote: post #116

I gave it two tries, GPT-4 was much better in both cases. Tried with two Leetcode questions. It came back with an empty response for one, and provided a worse code (O(n2) solutions when it can be done with linear time) for the other one. GPT-4 on the other hand provided a good answer for both questions. Also I guess the UI is buggy w.r.t code formatting, it things the following line is a code and switches to a code b…

I suggest you try enabling "Ignore search results" from the model dropdown for these types of questions. The web results can be distracting for the model for Leetcode-type questions.

I see you've had to suggest this a few times in this thread, and in my experience I would agree with the suggestion. I wonder if you can have a simple gpt model decide automatically when ignoring search results would improve the result and do it automatically.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#146

Post by pbhjpbhj »

make3 wrote: post #39
slowhadoken wrote: post #9

I love that Phind cites what it scrapes. This should be the obligation of all LLM. I always suggest people use it over ChatGPT.

What they're citing isn't what the LLM "scraped", it's what the retrieval model fed to the LLM. You're not guaranteed that it's what it actually used to give you the output, and it's also definitely not all the text that it used to get appropriate knowledge to generate the answer, as this is split over whatever millions of examples for the language and for human language in a non human-understandable way

A couple of times I've had the reference not include the detail being mentioned in the foregoing paragraph; the citations are still highly relevant, but it wasn't quite what I expected.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#147

Post by spmurrayzzz »

riku_iki wrote: post #95
Wytwwww wrote: post #90

Earlier quoted context omitted.

There is a performance conclusion in the title though.

That conclusion is based on benchmark with many examples in different tasks.

AFAIK they haven't released the dataset they fine-tuned on, so we can't be 100% there wasn't benchmark contamination. Agree that we definitely need more than N=1 to challenge the performance claims, but I still think its valid to call it out given how much benchmarking-gaming we've seen in this space.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#148

Post by pbhjpbhj »

lsaferite wrote: post #102
joshspankit wrote: post #43

Earlier quoted context omitted.

In fact, I’d argue that citation makes LLM better. Kind of a “think carefully” indicator. When LLMs are able to verify those citations independently it’s going to level up again by skyrocketing the objective truthiness.

Interestingly, I'd say that _not_ being able to give citations helps protect the LLM from copyright issues. That being said, I'm much prefer if the LLM could provide citations for every piece of information it was trained on and uses to provide an answer.

Citations are essential for me as I'm using Phind for work and can't rely on "trust me bro". It needs to confirm to my expectations or be confirmed in a couple of the citations that have trustworthy sources (eg are from known domains, well-cited journals, etc.).

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#149

Post by brucethemoose2 »

claytonjy wrote: post #113
brucethemoose2 wrote: post #18

> We can achieve up to 100 tokens per second single-stream while GPT-4 runs around 20 tokens per second at best. Is that with batching? If so, thats quite impressive. > certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4. Some of this is sampler tuning. Y'all should look at grammar based sampling ( https://git…

is that impressive? I was thinking 100 tok/s on an H100 is really slow considering LMDeploy claims 2000+ on an A100 and a large batch size.

Without batching, I was actually thinking that's kind of modest.

ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100:

https://github.com/turboderp/exllamav2#performance

I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my generation speed on a 3090.

100 tokens/s batched throughput (for each individual user) is much harder.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#150

Post by pbhjpbhj »

blago wrote: post #138

It failed for me at a much more basic level. I asked 5 different, and increasing explicit, variations of the following question: "Can you generate HTML and CSS for a JPG mockup I'm going to give you?" Each time it answered along the following lines: "Sure, here is how you can create HTML and CSS from a JPG mockup. Follow this process..." In my experience this never happens with GPT-4.

I've not seen that anywhere, ChatGPT does image input now? Do you have examples of the output from feeding it a JPEG?
Post reply on HN