Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

321–330 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#321

Post by Ghexor »

idonotknowwhy wrote: post #253
zoogeny wrote: post #112

I just spent a few minutes doing a comparison between Phind and GPT-4 for a very high-level question on a distributed job queue. I gave them both the same fairly vague sketch of a kind of system I would like to build. Here are my impressions: In the positives of Phind: * Phind was able, even eager, to recommend specific libraries relevant to the implementation. The recommendations matched my own research. GPT-4 takes…

This is a good point about GPT-4, it can intuit the "question behind the question" really well compared with other models. And it's been profoundly useful for me with the most random tasks I knew nothing about prior (like fixing a wall in my house), etc.

That's probably because OpenAi can train on the (succesfull) conversations we have with ChatGPT!

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#322

Post by pbronez »

Der_Einzige wrote: post #228
a_wild_dandan wrote: post #155

Earlier quoted context omitted.

They are. Moreover, the idea that AI companies are missing and/or not implementing this “obvious” tactic is hilarious. Folks, these approaches have profound consequences for training and inference performance. Y’all aren’t pointing out some low hanging fruit here, lol

Actually, yes I am pointing out low hanging fruit here. These approaches do not have "profound consequences" for inference or training performance. In fact, sentence transformer models run orders of magnitude more quickly. Performance penalties will be small. Also, I actually have several top NLP conference publications, so I'm not some charlatan when I say these things. I've actually physically used and seen these t…

Concur. LLM are still very young. We’re barely a year out from the ChatGPT launch. Everyone is iterating like mad. Several stealth companies working on new approaches with the potential to deliver performance leaps.

You ain’t seen nuthin’ yet…

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#323

Post by ErikBjare »

claytonjy wrote: post #162
lyjackal wrote: post #152

Earlier quoted context omitted.

I don't think they're saying they're doing batch size of 1, just giving performance expectations of user facing performance

I think they _are_ saying batch size 1, given that rushingcreek is OP.

Yes they are saying batch size 1 for the benchmarks, but they aren't doing batch size 1 in prod (obviously).

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#324

Post by SirMaster »

jiggawatts wrote: post #281
SirMaster wrote: post #271

Earlier quoted context omitted.

Even all of that is on the wrong track. There is nothing that I can see anywhere about controlling an ATV with the homekit accessory protocol.

Then you asked the wrong question. AFAIK Apple generally does not allow arbitrary remote control (headless mode) for security reasons — it could be used for spam automation!

They do though. Pyatv can do it (and home assistant is using pyatv since HA is python based) and commercial home automation systems like Crestron and Control4 can do it too.

Really I just need to get an LLM to port pyatv to C# for me I guess.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#325

Post by claytonjy »

ErikBjare wrote: post #323
claytonjy wrote: post #162

Earlier quoted context omitted.

I think they _are_ saying batch size 1, given that rushingcreek is OP.

Yes they are saying batch size 1 for the benchmarks, but they aren't doing batch size 1 in prod (obviously).

I don't think that is obvious. If your use case demands lowest latency at any cost, you might run batch size 1. I believe replit's new code model (announced about a month ago) runs at batch 1 in prod, for example, because code completions have to feel really fast to be useful.

With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#326

Post by rekubot »

notadev wrote: post #186

I use Phind daily, including the VSCode extension, and I love it. Much better than anything ChatGPT is able to come up and the code it generates requires little-to-no modification to work properly. Very big fan!

Far as I can tell it isn't possible to hook up the VSCode extension to the Phind model, only GPT-4. Do you know any different?

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#327

Post by brucethemoose2 »

claytonjy wrote: post #325
ErikBjare wrote: post #323

Earlier quoted context omitted.

Yes they are saying batch size 1 for the benchmarks, but they aren't doing batch size 1 in prod (obviously).

I don't think that is obvious. If your use case demands lowest latency at any cost, you might run batch size 1. I believe replit's new code model (announced about a month ago) runs at batch 1 in prod, for example, because code completions have to feel really fast to be useful. With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request…

I'm not sure about TensorRT, but in llama.cpp there are seperate kernals optimized for batching and single use inference. It makes a substantial difference.

I suppose one could get decent utilization by prompt processing one user while generating tokens for another.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#328

Post by riku_iki »

PoignardAzur wrote: post #317
riku_iki wrote: post #286

Earlier quoted context omitted.

that benchmark(HumanEval) is some public benchmark built by others.

That kind of benchmark is a lot more reliable for models published before the benchmarks; models published afterwards have more opportunity to "study to the test". That's especially a concern when a company explicitly uses its score on that benchmark as a marketing point.

sure, but it is the best thing we have.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#329

Post by ComputerGuru »

jeswin wrote: post #274
iandanforth wrote: post #201

Earlier quoted context omitted.

FWIW The default context window of GPT-4 via ChatGPT is about to change to 32k.

Given the number of times it just fails with large prompts on 32k contexts, I'm not sure if they'ready for this. In my experience, if you're consuming 20k+ tokens failure rate is more than 50%.

Well over 50%, at least via the api, for me.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#330

Post by emptysongglass »

riku_iki wrote: post #328
PoignardAzur wrote: post #317

Earlier quoted context omitted.

That kind of benchmark is a lot more reliable for models published before the benchmarks; models published afterwards have more opportunity to "study to the test". That's especially a concern when a company explicitly uses its score on that benchmark as a marketing point.

sure, but it is the best thing we have.

Well no we have the anecdotes of all the HN folks which I trust many, many times more than a benchmark.
Post reply on HN