ChatGPT vs. open source on harder tasks

Post by thomasahle » Thu, May 18, 2023, 8:30 PM UTC

andy_xor_andrew wrote: post #27

thomasahle wrote: post #25

Stuff like "Make sure to use real numbers, not fractions." seems like such a weird and fragile way to write a program. I thought the point of Microsoft Guidance was that we could specify a grammar or regular expression that the output must match?

I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules. I think regarding: "Make sure to use real numbers, not fractions." It would be simple using Guidance to create some kind of validation function, which guidance would invoke after it gets the response from the model. In this case you would check if the number is a real number or not. And…

Post by gajnadsgjoas » Thu, May 18, 2023, 8:42 PM UTC

hackernewds wrote: post #13

andy_xor_andrew wrote: post #6

I'm employed by Microsoft, but not in any relation to the authors of this tool or this post. First time I saw github.com/mirosoft/guidance was on the HN front page, actually :) I've been waiting for a tool exactly like guidance, something that lets me compose prompts the way I think about prompts. Langchain was too much of a headache for me. Playing with guidance, it feels like this is how programmatically interactin…

are we going to now invent a programming language that communicates with LLMs? I find that funny

Post by TheObviousOne » Thu, May 18, 2023, 8:47 PM UTC

Post by redblacktree » Thu, May 18, 2023, 8:51 PM UTC

bcrosby95 wrote: post #32

candiddevmike wrote: post #16

Earlier quoted context omitted.

I'm not an LLM, but I am a native English speaker and English is pretty awful for conveying information succinctly. The lengths you must go with prompts to ensure you're getting the right response is extremely similar to the lengths you go to when delegating something to someone or describing requirements to a project. I wonder if whatever language we end up using to "converse" with LLMs would have value within an or…

I prefix my longer prompts with glossaries. It lets me write more succinctly.

Post by icelancer » Thu, May 18, 2023, 8:52 PM UTC

doctoboggan wrote: post #9

furyofantares wrote: post #3

Earlier quoted context omitted.

I agree, I only use 4 because by far the thing that matters most for my usage is the quality of the output at this stage in the game. But 3.5 is generally available, fast, and costs a WHOLE LOT less for API users; those are real capabilities that matter though by "capable" you only meant the quality of the output. It's worth comparing to 3.5 also.

At first I was skeptical of the difference between 3.5 and 4 but there really is a large gap in the models understanding of complex issues between them. I make a habit of always selecting gpt4 but forgot to yesterday. Within 3 responses I could immediately tell I was interacting with a less capable model.