Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

121–130 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#121
> Show HN: Phind Model beats GPT-4 at coding

Does it? I don't see any evidence of this strong claim in your post, and I think it's quite deceptive how the only link is to a benchmark of open source models (which doesn't include GPT-4). I've tried Phind a few times in the past when it made equally strong claims and been somewhat unimpressed. (To be fair, comparing anything to GPT-4 is tough!) I think it would strengthen your position significantly to simply say that you're the best of all open-source models.

To be honest though I've been completely ruined by https://cursor.sh/; copying and pasting results back and forth from a web UI to my IDE is so painfully slow when you do it tens or hundreds of times that I don't think I would be able to go back. I'd be happy to try out a Phind extension that has similar UI/UX if you ever make one.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#122

Earlier quoted context omitted.

is that impressive? I was thinking 100 tok/s on an H100 is really slow considering LMDeploy claims 2000+ on an A100 and a large batch size.

We get 100 tokens a second with batch size 1. Those 2000+ figures are for large batches.

Ah, that's fair, and faster than any of the LMDeploy stats for batch size 1; nice work!

Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#123

So I gave it this prompt: > I need a typescript function which takes in an object with an id string property and a string message property, and also takes an array of search strings, and returns a mapping of search strings to matching message ids The response I got was close, but it assumed that each search string would match only one message, so it returned Record . I fed this to GPT-3.5 and it answered 10x faster w…

> I need a typescript function which takes in an object with an id string property and a string message property, and also takes an array of search strings, and returns a mapping of search strings to matching message ids

Your prompt is wrong. You want a function that takes an array of id/message objects, not an object.

It's quite impressive that GPT is just able to correct for that. As a human, I would first ask what you actually mean, because your prompt appears to be unclear.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#124
post #64

Earlier quoted context omitted.

Out of curiosity, why do you think the answer would be so simple and also completely untested?

Another curiosity, what do we estimate (if it's even possible) the context window of a human? Obviously an extremely broad question, and of course it must have some sort of decay factor... but... would be interesting to get a rule of thumb number in terms of token count. I can imagine its massive!

I don't think it's massive. In fact, since it's roughly equivalent to working memory, I suspect it's on the order of 100 tokens at most.

It's just that, unlike these AIs, we're capable of online learning.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#125
post #64

Earlier quoted context omitted.

Out of curiosity, why do you think the answer would be so simple and also completely untested?

Another curiosity, what do we estimate (if it's even possible) the context window of a human? Obviously an extremely broad question, and of course it must have some sort of decay factor... but... would be interesting to get a rule of thumb number in terms of token count. I can imagine its massive!

Human memory, in my limited understanding, doesn’t have the bifurcation of weights and context that LLMs do. It’s all a bit blurrier than that.

Something interesting that I heard from people trying to memorize things better is that memory “storage space” limits for people are essentially irrelevant. We’re limited by our learning and forgetting speeds. There’s no evidence of brains getting “full”.

Think of it like a giant warehouse of plants, with one employee. He can accept shipments (learning). He can take care of plants (remembering). Too long without care and they die (forgetting). The warehouse is big enough that it is not a limiting factor in how many plants he can keep alive. If it was 10x bigger it wouldn’t make a bit of difference.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#126
Been subscribed to Phind for 3 months now on 30€ per month, and constant outages made me unsubscribe this month. I did compare Phind and GPT-4 in the past, whenever Phind came out with these kind of articles, and after first question it was obvious Phind was nowhere near.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#127

Earlier quoted context omitted.

Still waiting for the day that medium term memory (token average pooling like in sentence transformers) becomes used for this. It's staring all of these companies in the face and apparently no one thinks to implement it.

I've been thinking along the same lines. The token window IMO should be a conceptual inverted pyramid, where there most recent tokens are retained verbatim but previous iterations are compressed/pooled more and more as the context grows. I'm sure there's some effort/research in this direction. It seems pretty obvious.

But some of the earlier tokens are also the most important ones, right? Like the instructions and rules you want it to follow.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#128

Been subscribed to Phind for 3 months now on 30€ per month, and constant outages made me unsubscribe this month. I did compare Phind and GPT-4 in the past, whenever Phind came out with these kind of articles, and after first question it was obvious Phind was nowhere near.

Sorry to hear that you didn't have a great experience. I'd love to chat further, my email is founders(at)phind.com

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#129
That bot of yours is second chat bot that claimed it can program or can help with programming. And that bot of yours is second one that utterly failed to provide me with an implementation of blocked clause decomposition in Haskell. I needed something, even the most slow version would do.

Yours' bot also tried to bullshit me about validity of its answer, just like the other one.

The difference? Your bot mentioned a paper on arxiv about the problem. But the paper (and I read it long time ago, of course) does not provide even pseudocode implementations of most of the algoithms mentioned there.

Color me not impressed.

As usual, bots like yours are not for when you need something new. If I have an idea, I cannot use any AI, including yours, for prototyping work.

It is expected as neural networks are interpolators, not extrapolators and for them to "extrapolate" one need to train them over the "extrapolation" area quite well.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#130
post #90

Earlier quoted context omitted.

There is a performance conclusion in the title though.

That conclusion is based on benchmark with many examples in different tasks.

From what I understand it's a single test suite? Of course I don't really mind the clickbait title that much, it's hard to attract attention otherwise.
Post reply on HN