121–130 of 179 posts
Earlier quoted context omitted.
If you show your task/prompt with an example I'll see if I can fix it and explain my steps. Are you using the function calling/tool use API?
Appreciate you being willing to help! It's pretty long, mind if I email/dm to you?
I have a personal writing app that uses the OpenAI models and this post is bang on. One of my learnings related to "Lesson 1: When it comes to prompts, less is more": I was trying to build an intelligent search feature for my notes and asking ChatGPT to return structured JSON data. For example, I wanted to ask "give me all my notes that mention Haskell in the last 2 years that are marked as draft", and let Chat GPT f…
Earlier quoted context omitted.
This is a great question! I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed. And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON. So I think there’s lots of room. One thing LangChain is doing now that solves something that IS ver…
It can be faster and more effective to fallback to a smaller model (gpt3.5 or haiku), the weakness of the prompt will be more obvious on a smaller model and your iteration time will be faster
The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…
> But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. Thank you. Seeing similar things. Clients are also seeing sticker shock on how much the big models cost vs. the output. That will all come down over time.
So will interest, as more and more people realise theres nothing "intelligent" about the technology, it's merely a Markov-chain-word-salad generator with some weights to improve the accuracy somewhat.
I'm sure some people (other than AI investors) are getting some value out of it, but I've found it to be most unsuited to most of the tasks I've applied it to.
The lessons I wanted from this article weren't in there: Did all of that expenditure actually help their product in a measurable way? Did customers use and appreciate the new features based on LLM summarization compared to whatever they were using before? I presume it's a net win or they wouldn't continue to use it, but more specifics around the application would be helpful.
Hey, OP here! The answer is a bit boring: the expenditure definitely has helped customers - in that, they're using AI generated responses in all their work flows all the time in the app, and barely notice it. See what I did there? :) I'm mostly serious though - one weird thing about our app is that you might not even know we're using AI, unless we literally tell you in the app. And I think that's where we're at with…
Earlier quoted context omitted.
> get it to say when it's not sure This is a function of the language model itself. By the time you get to the output, the uncertainty that is inherent in the computation is lost to the prediction. It is like if you ask me to guess heads or tails, and I guess heads, I could have stated my uncertainty (e.g. Pr [H] = .5) before hand, but in my actual prediction of heads, and then the coin flip, that uncertainty is lost…
Apparently it is possible to measure how uncertain the model is using logprobs, there's a recipe for it in the OpenAI cookbook: https://cookbook.openai.com/examples/using_logprobs#5-calcul... I haven't tried it myself yet, not sure how well it works in practice.
Earlier quoted context omitted.
Apparently it is possible to measure how uncertain the model is using logprobs, there's a recipe for it in the OpenAI cookbook: https://cookbook.openai.com/examples/using_logprobs#5-calcul... I haven't tried it myself yet, not sure how well it works in practice.
There’s a difference between certainty of the next token given the context and the model evaluation so far and certainty about an abstract reasoning process being correct given it’s not reasoning at all. These probabilities and stuff coming out are more about token prediction than “knowing” or “certainty” and are often confusing to people in assuming they’re more powerful than they are.
The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…
Earlier quoted context omitted.
Apparently it is possible to measure how uncertain the model is using logprobs, there's a recipe for it in the OpenAI cookbook: https://cookbook.openai.com/examples/using_logprobs#5-calcul... I haven't tried it myself yet, not sure how well it works in practice.
You can ask the model sth like: is xyz correct, answer with one word, either Yes or No. The log probs of the two tokens should represent how certain it is. However, apparently RLHF tuned models are worse at this than base models.