Ask HN: Best Practices for LLM Chatbot that references user account details?

#1

Post by olives » Wed, Jul 24, 2024, 3:12 PM UTC

What's the current best practice for including personalized user account details (e.g. for queries like "How much did I spend on this credit card last month?") into a LLM chatbot?

Specifically, assuming the user has already been authenticated properly, how is the user details retrieved, how is the data re-formatted, and how is the data ultimately fed into the LLM? This is a function in many popular banking chatbots, e.g. Bank of America's Erica, but there is scarce information about the actual implementation.

Re: Ask HN: Best Practices for LLM Chatbot that references user account details?

#2

Post by brianjking » Wed, Jul 24, 2024, 3:30 PM UTC

This is often called Function calling or tool use where you provide the LLM the ability to execute API calls to external tools.

Re: Ask HN: Best Practices for LLM Chatbot that references user account details?

#3

Post by olives » Thu, Jul 25, 2024, 1:51 AM UTC

brianjking wrote: post #2

This is often called Function calling or tool use where you provide the LLM the ability to execute API calls to external tools.

Is function calling the only way to do this? If the user account has some associated text, for example, historical medical files, is there a way to pass that text in such that the user could query "Write a short summary about my medical history?"

Re: Ask HN: Best Practices for LLM Chatbot that references user account details?

#4

Post by fanweixiao » Thu, Jul 25, 2024, 10:02 AM UTC

olives wrote: post #3

brianjking wrote: post #2

This is often called Function calling or tool use where you provide the LLM the ability to execute API calls to external tools.

Is function calling the only way to do this? If the user account has some associated text, for example, historical medical files, is there a way to pass that text in such that the user could query "Write a short summary about my medical history?"

You can create a function calling, the parameter is the user_id or a fingerprint of the user for backend. then, get all data from your database, returned as a JSON object, then, you will get the result.

The prompt will looks like:

""" you are a professional medical history recorder, write a short summary about the user's medical history, use tool_call when possible.

the current user_id is: abcd12345 """

your tool_call description should be set clearly, looks like:

""" get medical history by given user_id """