Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

161–170 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#161

Post by riku_iki »

spmurrayzzz wrote: post #147
riku_iki wrote: post #95

Earlier quoted context omitted.

That conclusion is based on benchmark with many examples in different tasks.

AFAIK they haven't released the dataset they fine-tuned on, so we can't be 100% there wasn't benchmark contamination. Agree that we definitely need more than N=1 to challenge the performance claims, but I still think its valid to call it out given how much benchmarking-gaming we've seen in this space.

I think you can bring contamination claim to every public benchmark results nowdays: models are trained on TBs of data crawled from internet, and there is no guarantee benchmark is not leaked in some way.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#162

Post by claytonjy »

lyjackal wrote: post #152
claytonjy wrote: post #122

Earlier quoted context omitted.

Ah, that's fair, and faster than any of the LMDeploy stats for batch size 1; nice work! Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?

I don't think they're saying they're doing batch size of 1, just giving performance expectations of user facing performance

I think they _are_ saying batch size 1, given that rushingcreek is OP.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#163

Post by popularonion »

shazar wrote: post #116

I gave it two tries, GPT-4 was much better in both cases. Tried with two Leetcode questions. It came back with an empty response for one, and provided a worse code (O(n2) solutions when it can be done with linear time) for the other one. GPT-4 on the other hand provided a good answer for both questions. Also I guess the UI is buggy w.r.t code formatting, it things the following line is a code and switches to a code b…

GPT-4 has ingested all of Leetcode, you can literally just type "leetcode 100 python" and it will regurgitate a response for you.

Only exception I found is with some of the Leetcode Premium questions, you might have to actually type in the problem statement, but it's still very likely that multiple solutions have been ingested from GitHub and elsewhere.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#164

Post by wizzwizz4 »

SirMaster wrote: post #58
jiggawatts wrote: post #44

Earlier quoted context omitted.

That’s because you’re asking it something too obscure that I would have at first assumed wasn’t even possible. “Make me a billionaire… I’m still poor! Bad AI!” You need to collaborate with the AI, use it to help with each small step of the problem, with input references provided. To a degree Phind can do the reference chasing for you, but it’s not magic .

It's definitely not impossible at least. Someone is doing it in python here: https://pyatv.dev/ GPT-4 actually sent me here: "Here is an example of a C# library that implements the HAP: CSharp.HomeKit ( https://github.com/brutella/hkhomekit ). You can use this library as a reference or directly use it in your project." Which, to no surprise based on my experiences with LLMs for programming does not exist and doesn't…

> Or I am just not good enough at prompting.

Or you're good enough at using your tools that you can do all the low-hanging fruit. LLMs excel at working around inadequate tooling, but (at least at the moment) they can't help you if you're trying to do something actually tricky and get stuck enough that no rubber duck can save you.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#165

Post by [deleted] »

ShakataGaNai wrote: post #139

"Python script to extract a list of all Elastic IP's from all regions, from multiple AWS accounts." ChatGPT4 gave me a solid answer hitting all the points I wanted. Phind din't get the account handling correct, didn't address regions, and didn't handle pagination. "Write a python based script that uses boto3 to query AWS Route53. It should print a list of every record for a given hosted zone ID." ChatGPT4 did exactly…

[deleted]

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#166

Post by __jonas »

ryanklee wrote: post #41
Racing0461 wrote: post #29

Earlier quoted context omitted.

As a user, i perfer getting the right response compared to the thing spitting out a link. (not saying phind is bad). Lets focus on getting llm right before nerfing it in its baby stages.

Who said anything about nerfing? Citation is just additive, no?

I find it often makes the responses worse when it's being pre-fed these search results, it was the case when I tried gpt-4 with web browsing enabled, and seems to be the case with this, since even the person from the Phind team in this thread pointed out that turning this feature off improves performance for some tasks:

https://news.ycombinator.com/item?id=38089888

https://news.ycombinator.com/item?id=38090442

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#167

Post by facu17y »

rushingcreek wrote: post #144
ShakataGaNai wrote: post #139

"Python script to extract a list of all Elastic IP's from all regions, from multiple AWS accounts." ChatGPT4 gave me a solid answer hitting all the points I wanted. Phind din't get the account handling correct, didn't address regions, and didn't handle pagination. "Write a python based script that uses boto3 to query AWS Route53. It should print a list of every record for a given hosted zone ID." ChatGPT4 did exactly…

Just worked well for me: https://www.phind.com/search?cache=g9y2uizgjwcn378aovb65v92 . We do have issues with consistency sometimes -- please try regenerating if that is the case.

"We do have issues with consistency sometimes" That's a strange statement. Having issues with consistency means that sometimes the output is wrong. What does it mean to have issues with consistency sometimes ? You're either consistent or you're not.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#168

Post by shmoogy »

JoshGlazebrook wrote: post #63
BugsJustFindMe wrote: post #25

> You can now get high quality answers for technical questions in 10 seconds instead of 50. ChatGPT 4 does not take 50 seconds to answer, so I don't understand this comparison.

ChatGPT4 is more often than not noticeably slow enough that I question why I pay for it.

Sometimes it's insanely quick - like gpt3,5 turbo or a cached answer or something.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#169

Post by bigfudge »

theWreckluse wrote: post #54
joshspankit wrote: post #42

Earlier quoted context omitted.

Not my experience at all. Are you counting the entire answer in your time? If so, consider adding one of the “just get to the point” prompts. GPT4’s defaults have been geared towards public acceptance through long-windedness which is imo entirely unnecessary when using it to do functional things like scp a file.

LOL, it’s not just for “public acceptance”. Look up Chain of Thought. Asking it to get to the point typically reduces the accuracy.

You are being snarky but are right. I have scripts set up to auto summarise expansive answers. I wish I could build this into the ChatGPT ui though.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#170

Post by rushingcreek »

facu17y wrote: post #167
rushingcreek wrote: post #144

Earlier quoted context omitted.

Just worked well for me: https://www.phind.com/search?cache=g9y2uizgjwcn378aovb65v92 . We do have issues with consistency sometimes -- please try regenerating if that is the case.

"We do have issues with consistency sometimes" That's a strange statement. Having issues with consistency means that sometimes the output is wrong. What does it mean to have issues with consistency sometimes ? You're either consistent or you're not.

There's a difference between models that are incompetent and aren't capable of getting the right answers ever and models that are capable of getting the right answer but may not do so every time. The Phind Model is in the latter camp.

Consistency issues can be caused by a wide range of factors from inference hyperparameters to prompting.

Post reply on HN