Such an AWESOME time to be a programmer!!
Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
61–70 of 358 posts
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#62Didn't work fine when I asked it a design question: the code and API it used is not correct. GPT-4 did a better job. https://www.phind.com/search?cache=ay8rx37gq8oy3z7uixftlqkt https://chat.openai.com/share/a3a91dcc-a91a-4b04-8afd-40bd1a...
Thanks for sharing the links, we'll investigate this example.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#63> You can now get high quality answers for technical questions in 10 seconds instead of 50. ChatGPT 4 does not take 50 seconds to answer, so I don't understand this comparison.
ChatGPT4 is more often than not noticeably slow enough that I question why I pay for it.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#64> it supports up to 16k tokens > Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0] This is wild to me. The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very imp…
Still waiting for the day that medium term memory (token average pooling like in sentence transformers) becomes used for this. It's staring all of these companies in the face and apparently no one thinks to implement it.
Out of curiosity, why do you think the answer would be so simple and also completely untested?
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#65> it supports up to 16k tokens > Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0] This is wild to me. The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very imp…
Token window size is being virtualized with the like of MemGPT, so its effect will diminish.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#66So I gave it this prompt:
> I need a typescript function which takes in an object with an id string property and a string message property, and also takes an array of search strings, and returns a mapping of search strings to matching message ids
The response I got was close, but it assumed that each search string would match only one message, so it returned Record. I fed this to GPT-3.5 and it answered 10x faster with the correct return type.
This is a slightly tricky example, because it requires the model to infer that multiple message matches are possible. But I think that it’s interesting that ChatGPT nailed it despite not using any chain of thought.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#67I tried this question and GPT4 did way way better to getting closer to a final answer. Phind was horribly wrong. I can't help but think something seems off with your eval given just how badly Phind did on this.
I want to make an interactive plot in Colab where I can show
X axis is interest rate of a 15 year mortgage. Y axis is the relative advantage of buying a house vs. renting in terms of total net worth at 15 years.
Assume a monthly budget for renting + investing or buying a house of 10k
Plot different lines for a few different market returns.
Make a slider that controls the total size of the loan.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#68In my experience, Phind is not as good as GPT4, but it's by far the second best LLM for programming. I find that tremendously impressive considering they are competing against the whole world for that title right now.
I agree with the assessment about consistency being its major flaw. While with GPT4 I can continue a conversation for quite long, Phind easily looses the required context. Perhaps it has to do with summarization capabilities, or messing with the context window has these types of side effects.
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#69Well, neither GPT4 or this Phind model where able to answer my torture test:
"Write amaranth code that can be used to control the readout of a frame from a kodak CCD with 4096 columns and 2048 rows."
Which yes, is missing a lot of detail (you could/I have feed/fed in a datasheet).
But Phind goes off on using pyserial (?!), and GPT4 assumes amaranth is a hypothetical CCD control library and makes a useless class control CCD using the hypothetical library.
Edit - Phind at least acknowledged that amaranth exists, unlike GPT4 with this prompt: "Write amaranth code that can be used to control the readout of a frame from a kodak CCD using an lattice FPGA with 4096 columns and 2048 rows. Assume the design will be hooked up to a larger litex SoC "
Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
#70Earlier quoted context omitted.
Still waiting for the day that medium term memory (token average pooling like in sentence transformers) becomes used for this. It's staring all of these companies in the face and apparently no one thinks to implement it.
Out of curiosity, why do you think the answer would be so simple and also completely untested?
Another curiosity, what do we estimate (if it's even possible) the context window of a human? Obviously an extremely broad question, and of course it must have some sort of decay factor... but... would be interesting to get a rule of thumb number in terms of token count. I can imagine its massive!