Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

phind.com

Post reply on HN
Print view View original

231–240 of 358 posts

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#231

Post by donmcronald »

rushingcreek wrote: post #216
ojosilva wrote: post #214

Awesome model from a quick run-through comparison, it's comparable in results to GPT-4 with web search and references as a plus, but runs faster. Two small nitpicks: - Dark mode is hard to read, the answer text font has too much weight and brightness which makes it hard to read long paragraphs of non-code text. Light mode is obviously too bright overall, but it's already nighttime where I'm at so maybe tomorrow at no…

Thanks for the feedback. We also support GPT-4 as an answering model so users can pick and choose what's best for their use case, but we recommend the Phind Model for the majority of users.

Why is there an 8x difference in price-per-search between Plus and Pro?

I always shy away from stuff like this because I view it as one of two things. Either I'm getting ripped off if I pay for Plus, because 8x the cost to me means your margin is huge, or I'm getting subsidized by you with the Pro version which means I can't rely on it lasting long term.

I also dislike daily limits for search. My search usage isn't uniform day-to-day. I might go most of the month without searching for anything and then do a ton of searching over 2-3 days when I'm trying to learn something. So I'll be idle most of the month and then not have enough searches on the days I actually want to use it.

I prefer the model used by a lot of pre-paid services. Let me deposit a chunk of money (ex: $20-50 minimum) and charge me per search until my money is gone. That way I'm not "losing out" if I don't use it every day and I can "burst" as high as I want when I'm trying to learn something.

If the pricing is based on a certain amount of loss (on my side) from the use-it-or-lose it model, I don't like that. I want simple, fair pricing, not a complex pricing scheme where the primary purpose is to get me to overpay for my usage.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#232

Post by dazhbog »

I found phind v7 to be closer to gpt3.5. First answer was great, but quickly started repeating mistakes from previous prompts. I also felt gpt4 understood better the constraints of the problem.

Still a massive thumbs up to the phind team. Impressive stuff!

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#233

Post by rushingcreek »

donmcronald wrote: post #231
rushingcreek wrote: post #216

Earlier quoted context omitted.

Thanks for the feedback. We also support GPT-4 as an answering model so users can pick and choose what's best for their use case, but we recommend the Phind Model for the majority of users.

Why is there an 8x difference in price-per-search between Plus and Pro? I always shy away from stuff like this because I view it as one of two things. Either I'm getting ripped off if I pay for Plus, because 8x the cost to me means your margin is huge, or I'm getting subsidized by you with the Pro version which means I can't rely on it lasting long term. I also dislike daily limits for search. My search usage isn't u…

Phind Plus is $15/month and Phind Pro is $30/month. There is a 2x price difference, not an 8x difference. And Phind Pro comes with (virtually) unlimited GPT-4 uses.

We understand that the incentives of setting daily limits for search aren't great, which is why the Phind model is unlimited for free. GPT-4, however, is unfortunately too expensive for us not to charge past a certain usage threshold.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#234

Post by donmcronald »

Racing0461 wrote: post #29
slowhadoken wrote: post #9

I love that Phind cites what it scrapes. This should be the obligation of all LLM. I always suggest people use it over ChatGPT.

As a user, i perfer getting the right response compared to the thing spitting out a link. (not saying phind is bad). Lets focus on getting llm right before nerfing it in its baby stages.

Give me the citations every day of the week. The source of information matters. For example, I don't rely on any ZFS info or opinions I find online if I can't verify it came from a contributor or highly reputable person that has a lot of experience with ZFS.

If you want to show the warts of all these LLMs, ask it about ZFS if you know enough to spot the commonly parroted misinformation that plagues the internet.

IMHO, these systems look super useful if they're citing sources and they're worthless without.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#235

Post by jiggawatts »

SirMaster wrote: post #58
jiggawatts wrote: post #44

Earlier quoted context omitted.

That’s because you’re asking it something too obscure that I would have at first assumed wasn’t even possible. “Make me a billionaire… I’m still poor! Bad AI!” You need to collaborate with the AI, use it to help with each small step of the problem, with input references provided. To a degree Phind can do the reference chasing for you, but it’s not magic .

It's definitely not impossible at least. Someone is doing it in python here: https://pyatv.dev/ GPT-4 actually sent me here: "Here is an example of a C# library that implements the HAP: CSharp.HomeKit ( https://github.com/brutella/hkhomekit ). You can use this library as a reference or directly use it in your project." Which, to no surprise based on my experiences with LLMs for programming does not exist and doesn't…

In a sense you’re asking it the wrong questions. It’s a bit like asking Google “my PC crashed, how do I fix!?” and then expecting something specific to a rare issue in the first hit.

Assuming a C# library even exists for what you’re doing (maybe not!) then still the best use of AI is to troubleshoot specific issues given an almost working piece of code as input.

Ask it to explain why something doesn’t work instead of asking it to do your job for you wholesale.

PS: GPT 4 (you are using the best coding AI, right? Right?) can get you going quickly:

“There are several libraries available for controlling Apple HomeKit from C#. One such library is *HapSharp* ². It is a .NET implementation of the HomeKit Accessory Server that allows you to create your own custom HomeKit accessory on a Raspberry Pi, Mac computer, or any other platform that can run Mono ².

Another option is *HomeKit* ¹. It is a native C# library for Apple's HomeKit Accessory Protocol. However, it is not a complete implementation and does not work ¹.

I hope this helps!

Source: Conversation with Bing, 31/10/2023 (1) netonjm/HapSharp: HomeKit Accessory Server .Net bridge! - GitHub. https://github.com/netonjm/HapSharp. (2) GitHub - ppumkin/HomeKit: Native C# Libary for Apple's HomeKit .... https://github.com/ppumkin/HomeKit. (3) homekit-accessory-protocol · GitHub Topics · GitHub. https://github.com/topics/homekit-accessory-protocol?o=asc&s...

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#236

Post by rushingcreek »

SirMaster wrote: post #28

I tried this, but I still have yet to get any LLM to answer me a programming question (that actually works) that I actually want to solve. Basically: "How can I send network control commands to an AppleTV in C#" They always make up some nonexistent library or gives an example using some nonexistent API.

Interesting, I seem to have gotten a decent answer: https://www.phind.com/search?cache=avbridtm69ejk8pdqpx8hcnf

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#237

Post by donmcronald »

rushingcreek wrote: post #233
donmcronald wrote: post #231

Earlier quoted context omitted.

Why is there an 8x difference in price-per-search between Plus and Pro? I always shy away from stuff like this because I view it as one of two things. Either I'm getting ripped off if I pay for Plus, because 8x the cost to me means your margin is huge, or I'm getting subsidized by you with the Pro version which means I can't rely on it lasting long term. I also dislike daily limits for search. My search usage isn't u…

Phind Plus is $15/month and Phind Pro is $30/month. There is a 2x price difference, not an 8x difference. And Phind Pro comes with (virtually) unlimited GPT-4 uses. We understand that the incentives of setting daily limits for search aren't great, which is why the Phind model is unlimited for free. GPT-4, however, is unfortunately too expensive for us not to charge past a certain usage threshold.

Plus costs $0.016 per search and Pro costs $0.002 per search.

https://www.phind.com/search?cache=wgyz13tg4jkbl9pklptmpds5

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#238

Post by CSMastermind »

I was just testing to see the comparison and ran into a message saying I was out of GPT-4 queries, despite having deliberately selected Phind as my model.

Now I'm confused if the results I was seeing really were from a different model than GPT-4 or not.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#239

Post by riku_iki »

Wytwwww wrote: post #130
riku_iki wrote: post #95

Earlier quoted context omitted.

That conclusion is based on benchmark with many examples in different tasks.

From what I understand it's a single test suite? Of course I don't really mind the clickbait title that much, it's hard to attract attention otherwise.

I think it is valid criticism that that HumanEval benchmark is not completely representative, they also say it in the post.

Re: Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context

#240

Post by spmurrayzzz »

riku_iki wrote: post #161
spmurrayzzz wrote: post #147

Earlier quoted context omitted.

AFAIK they haven't released the dataset they fine-tuned on, so we can't be 100% there wasn't benchmark contamination. Agree that we definitely need more than N=1 to challenge the performance claims, but I still think its valid to call it out given how much benchmarking-gaming we've seen in this space.

I think you can bring contamination claim to every public benchmark results nowdays: models are trained on TBs of data crawled from internet, and there is no guarantee benchmark is not leaked in some way.

With respect to the pretraining data, its true that we're probably SOL there in terms of verification. But for fine-tuning, they could still publish the dataset and see if others can reproduce their results as well as audit for contamination.

If we're comparing benchmark deltas between different fine-tuned variants that share the same base models, that seems like the bare minimum we should expect to come along with performance claims.

Post reply on HN