Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Agents in Haystack: Make LLMs resolve complex tasks

haystack.deepset.ai

Post reply on HN
Print view View original

21–30 of 106 posts

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#21

Post by nico »

Art9681 wrote: post #12
jcims wrote: post #7

From the Github project readme >Agent: (since 1.15) An Agent is a component that is powered by an LLM, such as GPT-3. It can decide on the next best course of action so as to get to the result of a query. It uses the Tools available to it to achieve this. While a pipeline has a clear start and end, an Agent is able to decide whether the query has resolved or not. It may also make use of a Pipeline as a Tool. Emphasis…

I've been tinkering with LangChain for a few days and I agree. Is there a resource that collects the agents so we can experiment? I'd love to see an aggregated list of the most impressive agents and use cases if anyone knows of any. This stuff is the future of computing no doubt.

There is a LangChainHub mentioned in their docs, but the repo for it seems dead.

Is there any sort of marketplace/AppStore for agents/tools/plug-ins for LLMs via LangChain? Or some other library like haystack?

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#22

Post by Ozzie_osman »

lsy wrote: post #8

Is there some way of holding the LLM response to a given prompt constant? It sounds like a lot of this relies on the LLM getting the right answer in sequence, so I'm guessing they do something like keep the temperature at 0? Otherwise you are going to wind up with possibly different behavior run-to-run. And even if they do have something like the above, don't we end up with potentially breaking changes once models ar…

Even with temperature 0 and the same model, the models are slightly non-deterministic and may diverge with the same input.

You probably don't want to treat them as deterministic (at least, not now). But there are many applications where slightly non-deterministic behavior is OK.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#23

Post by Ozzie_osman »

If you (like me) were wondering how these works, the LLM is given a prompt like:

  Answer the following questions as best you can. You have access to the following tools:
  Search: Use this to search the internet.
  Calculator: Use this to do math.

  Use the following format:
  Question: the input question you must answer
  Thought: you should always think about what to do
  Action: the action to take, should be one of [{tool_names}]
  Action Input: the input to the action
  Observation: the result of the action
  ... (this Thought/Action/Action Input/Observation can repeat N times)
  Thought: I now know the final answer
  Final Answer: the final answer to the original input question

  Question: What is the age of the president of Egypt squared?
  Thought:
To which the LLM will generate a completion like:

  Thought: I need to find the age of the president of Egypt.
  Action: Search
  Action Input: Age of president of Egypt
  Observation:
At which point, the code (langchain, haystack, etc) will parse out the requested tool (Search) and input (Age of president of Egypt), and then call the right tool or API, then append the output of that action into the prompt.

This all happens in a loop, at each step, the LLM is given the entire past prompt history, and given the opportunity to do a completion to choose the next tool and input to the next tool, after which the code parses those out, executes the tool, and repeats until the LLM decides it has the final answer and returns.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#24

Post by Cyphase »

amrrs wrote: post #9
lsy wrote: post #8

Is there some way of holding the LLM response to a given prompt constant? It sounds like a lot of this relies on the LLM getting the right answer in sequence, so I'm guessing they do something like keep the temperature at 0? Otherwise you are going to wind up with possibly different behavior run-to-run. And even if they do have something like the above, don't we end up with potentially breaking changes once models ar…

There is a library called guardrails . I've not played with it extensively but that seems to address issues like this one.

https://shreyar.github.io/guardrails/

https://github.com/shreyar/guardrails

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#25

Post by madisonmay »

wickoff wrote: post #19
yosito wrote: post #10

I haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They may get to that at some point in the article, but the page eventually breaks for me on mobile and I can't read past the first code block.

I have the same exact problem with people. Being inaccurate is a feature, not a bug.

Imperfect systems are still useful, and any sufficiently complex system is imperfect.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#26

Post by Ozzie_osman »

Ozzie_osman wrote: post #23

If you (like me) were wondering how these works, the LLM is given a prompt like: Answer the following questions as best you can. You have access to the following tools: Search: Use this to search the internet. Calculator: Use this to do math. Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [{tool_name…

(also tbc I took these example prompts from LangChain.. not sure if Haystack uses different prompts (LangChain actually has a bunch of versions, this is probably the easiest one)

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#27

Post by teruakohatu »

wickoff wrote: post #19
yosito wrote: post #10

I haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They may get to that at some point in the article, but the page eventually breaks for me on mobile and I can't read past the first code block.

I have the same exact problem with people. Being inaccurate is a feature, not a bug.

It is different with LLMs. Most people can give a level of uncertainty along with an answer, and often do. LLMs can't, and worse, are trained to put an emphasis on the prompts. Humans are often trained to be skeptical of prompts.

If I said, "the moon is made of cheese. What type of cheese do you think it is?" most humans would automatically object, but with LLMs you can usually craft a prompt that would get it to answer such a silly question.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#28

Post by computerex »

lsy wrote: post #8

Is there some way of holding the LLM response to a given prompt constant? It sounds like a lot of this relies on the LLM getting the right answer in sequence, so I'm guessing they do something like keep the temperature at 0? Otherwise you are going to wind up with possibly different behavior run-to-run. And even if they do have something like the above, don't we end up with potentially breaking changes once models ar…

The output is stochastic, so if the response doesn't decode to your format you can try the request again. Gpt3.5-turbo is pretty good at tool selection and use, but it often messes up with difficult tasks. gpt-4 is on another level when it comes to tool use. It is very reliable in my testing.

You ofc can't guarantee the output so defensive programming, retries are a must in my opinion. We are all learning how to work with this technology.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#29

Post by Spivak »

teruakohatu wrote: post #27
wickoff wrote: post #19

Earlier quoted context omitted.

I have the same exact problem with people. Being inaccurate is a feature, not a bug.

It is different with LLMs. Most people can give a level of uncertainty along with an answer, and often do. LLMs can't, and worse, are trained to put an emphasis on the prompts. Humans are often trained to be skeptical of prompts. If I said, "the moon is made of cheese. What type of cheese do you think it is?" most humans would automatically object, but with LLMs you can usually craft a prompt that would get it to ans…

I mean it kinda can. Here's the full prompt. I have no idea about aspartame, I just picked something that it's definitely not sure about.

    Answer with a JSON object of the form {"confidence": $>, "en": $>}.
    User: What is 2 + 2? Bot: {"confidence": "very", "en": "2 + 2 is 4"}
    User: Is aspartame healthy? Bot: {"confidence": "somewhat", "en":
    "Aspartame has not yet been found to have any adverse effects on
    humans."} User: Who won the war on 1812? Bot:

    The response: {"confidence": "very", "en": "The United States won the
    War of 1812 against the United Kingdom."}
Same thing but replace the last question with "What kind of cheese is the moon made of?"

    The response: {"confidence": "very low", "en": "I'm not sure, but I
    don't think the moon is made of cheese."}
How about "Is the economic system of communism viable long term?"

    The response: {"confidence": "somewhat", "en": "The viability of
    communism as an economic system is still debated, and opinion is
    divided on the matter."}

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#30

Post by ramraj07 »

PostOnce wrote: post #18
ramraj07 wrote: post #16

Earlier quoted context omitted.

Gpt-4 does a reasonable job citing things. It can’t cite every paper out there but definitely the well cited ones.

Does it cite papers that don't exist, or cite papers when the paper it cites doesn't actually contain the information being cited? I would bet it does, at least some percent of the time.

The latter, yes. Interestingly I’m not surprised at all. This is what many researchers themselves do lol. I never take a reference at face value from any human being and I apply the same standard to gpt-4 as well. But all its references are real. Just 20-40% of time it might not exactly say the same as what I asked it for (though it’s related, and mostly there).
Post reply on HN