Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

51–60 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#51

Post by londons_explore »

joshspankit wrote: post #42
bethekind wrote: post #34

Earlier quoted context omitted.

Recently I've used gpt 4 and yes it does take up to a minute even for easy questions. I've asked it how to scp a file on Windows 11 and it'll take a minute to tell me all the options possible. If this takes 1/5th the time for equivalent questions, I'd consider switching

Not my experience at all. Are you counting the entire answer in your time? If so, consider adding one of the “just get to the point” prompts. GPT4’s defaults have been geared towards public acceptance through long-windedness which is imo entirely unnecessary when using it to do functional things like scp a file.

The words "briefly" or "without explanation" work well.

By keeping the prompt short, it starts generating output quicker too.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#52

Post by danenania »

SirMaster wrote: post #28

I tried this, but I still have yet to get any LLM to answer me a programming question (that actually works) that I actually want to solve. Basically: "How can I send network control commands to an AppleTV in C#" They always make up some nonexistent library or gives an example using some nonexistent API.

I’m working on an open source, terminal-based AI coding tool that is designed specifically for more complex, multi-iteration tasks and features. I think it could likely do a good job on this task.

I’m using it personally every day and while it still needs more work and polish, I’m finding it much better than ChatGPT or any other tools I’ve tried for bigger and more difficult tasks.

Please let me know if you (or anyone else reading this) would be interested to try a late alpha/early beta version: dane@envkey.com

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#53

Post by jiggawatts »

seidleroni wrote: post #12

The results I get are so-so. The rubric I use to evaluate coding LLM's is to ask it to create a Python script that determines if the contents of a given directory have been changed since the last time the script was run. This should be done recursively and handle files being added, removed, or modified and be based off the contents of the files and not the timestamps. When I ask it as one statement it performed ok, b…

This is a problem that human programmers screw up… regularly.

E.g.: the efficient and robust file change monitoring on Windows is to read the NTFS change journal. For a single process lifetime there are other change notification APIs as well. Most software does neither and is either very slow or misses changes…

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#54

Post by theWreckluse »

joshspankit wrote: post #42
bethekind wrote: post #34

Earlier quoted context omitted.

Recently I've used gpt 4 and yes it does take up to a minute even for easy questions. I've asked it how to scp a file on Windows 11 and it'll take a minute to tell me all the options possible. If this takes 1/5th the time for equivalent questions, I'd consider switching

Not my experience at all. Are you counting the entire answer in your time? If so, consider adding one of the “just get to the point” prompts. GPT4’s defaults have been geared towards public acceptance through long-windedness which is imo entirely unnecessary when using it to do functional things like scp a file.

LOL, it’s not just for “public acceptance”. Look up Chain of Thought. Asking it to get to the point typically reduces the accuracy.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#55

Post by passion__desire »

rushingcreek wrote: post #33
eoinboylan wrote: post #31

Ran a quick test with a Rust async code snippet that contains an error. Compared with GPT-4 its gives a far clearer solution, with linked sources to learn more! Super impressive!

Amazing, that's great to hear.

Is it possible to output all steps of solutions in a single copyable block? I don't want to copy 4 separate blocks.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#56

Post by accrual »

> it supports up to 16k tokens

> Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0]

This is wild to me.

The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very impressive, but imagine it as it becomes more like an actual pair programmer and remembers all the various things it's learned and worked on with you in the past.

[0] https://huggingface.co/docs/transformers/main/model_doc/llam...

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#58

Post by SirMaster »

jiggawatts wrote: post #44
SirMaster wrote: post #28

I tried this, but I still have yet to get any LLM to answer me a programming question (that actually works) that I actually want to solve. Basically: "How can I send network control commands to an AppleTV in C#" They always make up some nonexistent library or gives an example using some nonexistent API.

That’s because you’re asking it something too obscure that I would have at first assumed wasn’t even possible. “Make me a billionaire… I’m still poor! Bad AI!” You need to collaborate with the AI, use it to help with each small step of the problem, with input references provided. To a degree Phind can do the reference chasing for you, but it’s not magic .

It's definitely not impossible at least.

Someone is doing it in python here:

https://pyatv.dev/

GPT-4 actually sent me here:

"Here is an example of a C# library that implements the HAP: CSharp.HomeKit (https://github.com/brutella/hkhomekit). You can use this library as a reference or directly use it in your project."

Which, to no surprise based on my experiences with LLMs for programming does not exist and doesn't seem to have ever existed.

I get that they aren't magic, but I guess I am just bad at trying to use LLMs to help in my programming. Apparently all I do are obscure things or something. Or I am just not good enough at prompting. But I feel like that's also a reflection of the weakness of an LLM in that it needs such perfect and specific prompting to get good answers.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#59

Post by xmprt »

iillexial wrote: post #47

Didn't work fine when I asked it a design question: the code and API it used is not correct. GPT-4 did a better job. https://www.phind.com/search?cache=ay8rx37gq8oy3z7uixftlqkt https://chat.openai.com/share/a3a91dcc-a91a-4b04-8afd-40bd1a...

The GPT-4 answer is only better in so far as it uses RunTransaction. I don't know why it's trying to loop through the stores and then running the i'th operation on that store when it could have just had the store referenced in the operation instead of passing it as a parameter. And then it's also creating a new client for each transaction which seems wrong (to be fair I'm not familiar with Firestore so maybe this is idiomatic).

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#60

Post by Der_Einzige »

accrual wrote: post #56

> it supports up to 16k tokens > Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0] This is wild to me. The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very imp…

Still waiting for the day that medium term memory (token average pooling like in sentence transformers) becomes used for this. It's staring all of these companies in the face and apparently no one thinks to implement it.
Post reply on HN