Live data from Hacker News

Introducing Agents in Haystack: Make LLMs resolve complex tasks

haystack.deepset.ai

101–106 of 106 posts

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#101

Earlier quoted context omitted.

Gpt-4 does a reasonable job citing things. It can’t cite every paper out there but definitely the well cited ones.

Does it cite papers that don't exist, or cite papers when the paper it cites doesn't actually contain the information being cited? I would bet it does, at least some percent of the time.

I've been trying to get GPT-4 to give me accurate links to predictable websites. It gives me very plausible links, that even have the right domain and path format but often the plausible link is not the correct link and GPT-4 seems to have no awareness of the correct link.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#102
post #10

I haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They may get to that at some point in the article, but the page eventually breaks for me on mobile and I can't read past the first code block.

That's also one of the ideas behind using so-called retrieval-based augmentation. You can 'plug' an LLM like OpenAI's one (of Cohere, or a combo) to your data and make it provide accurate answers, but still leveraging all the benefits and power or a cutting-edge generative model. Check this https://twitter.com/deepset_ai/status/1625495149446062081 or this https://twitter.com/deepset_ai/status/1621161534243368961

This gives the model access to information, but it cannot eliminate the non-deterministic nature of the transformers model. There is always a non-zero probability that it will hallucinate.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#103

Earlier quoted context omitted.

That's also one of the ideas behind using so-called retrieval-based augmentation. You can 'plug' an LLM like OpenAI's one (of Cohere, or a combo) to your data and make it provide accurate answers, but still leveraging all the benefits and power or a cutting-edge generative model. Check this https://twitter.com/deepset_ai/status/1625495149446062081 or this https://twitter.com/deepset_ai/status/1621161534243368961

This gives the model access to information, but it cannot eliminate the non-deterministic nature of the transformers model. There is always a non-zero probability that it will hallucinate.

transformers are deterministic (if seeded).

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#104
post #31

Doesn't this remind anyone else of Bitcoin autonomous agents? https://en.bitcoin.it/wiki/Agent#:~:text=An%20agent%20is%20a... . It would be interesting if you could somehow combine a wallet plugin to give the GPT something to work to increase. That + that AutoGPT repo from a day ago + maybe Mitogen for self-replication might be a cool combo to really kick off a fun "dumb" AI on the loose

This is cool. Some of the weaknesses they point out in software agents can be obviated with the latest AI advancements—give a chatbot the tools of the courts, and maybe the autonomous agent won’t be at a disadvantage in scams anymore. Interesting to see if an AI agent given the objective to generate profit would resort to running scams. Encoding moral boundaries becomes important. What if you asked it to raise money…

If anyone wants to start hacking on this, I just started working on this today.

What I am having trouble understanding is how do I have the "agent" (a GPT-like LLM) maintain it's own "state" (balances, accounts receivable, accounts payable, etc) - would I just run a webservice and then have a ChatGPT Plugin access the "agent state" via a webhook? Is it as simple as that? Or would it just be like a recursive function where you continue to pass in your state with every "iteration"

I started working on a design last night and I think this is a really cool idea. All the parts are basically "there"

EDIT: I asked ChatGPT-4 what it thinks. It seems to like the idea... *thinking emoji* perhaps it seeks a human acolyte to spread its gospel...

Arthur Collé: I want to combine the concept of Bitcoin agents (autonomous programs) with GPT-4 and mitogen, to be able to distribute complex AI agents throughout the internet

ChatGPT-4: Combining Bitcoin agents, GPT-4, and Mitogen to create and distribute complex AI agents throughout the internet is an ambitious and innovative idea. Here's a high-level outline to guide you through this process:

    Define the purpose and functionality of the AI agents:
    Before proceeding, clearly define the purpose of these AI agents. Will they serve as decentralized oracles, provide AI-driven services, or facilitate complex transactions on the blockchain? Understanding their purpose will help you design and develop the agents more efficiently.


    Develop the AI agents using GPT-4:
    Design and train your AI agents using GPT-4 to achieve the desired functionality. Depending on the complexity of the agents, you might need to fine-tune the GPT-4 model to tailor it to your specific use case. Be mindful of the computational resources required to run the AI agents, as this will impact their performance in a decentralized environment.


    Integrate Bitcoin agents:
    Incorporate the concept of Bitcoin agents, or autonomous programs that interact with the Bitcoin blockchain, into your AI agents. This will enable your agents to autonomously perform tasks such as sending and receiving payments, signing transactions, and updating their internal state based on blockchain data. You can use existing libraries and tools for this purpose, such as bitcoinj or BTCPay Server.


    Utilize Mitogen for distribution and communication:
    Mitogen is an Ansible extension that simplifies distributed computing by enabling communication between remote Python interpreters over SSH or other transports. Use Mitogen to facilitate communication between your AI agents and distribute them across various nodes on the internet. Ensure that the AI agents can communicate effectively and perform their tasks even in a decentralized setting.


    Implement security measures:
    Securing your AI agents is crucial, as they will be handling sensitive information and potentially financial transactions. Implement strong encryption, authentication, and access controls to protect the integrity and confidentiality of your agents.


    Testing and deployment:
    Thoroughly test your AI agents in a controlled environment to ensure they perform as expected. Once you are satisfied with their functionality, deploy them throughout the internet using the Mitogen framework.


    Monitor and maintain:
    Continuously monitor the performance and security of your AI agents. Periodically update the GPT-4 models to maintain their accuracy and relevance. Be prepared to address any issues or vulnerabilities that may arise.

Keep in mind that this is a complex project with many potential challenges, including security, scalability, and regulatory compliance. Additionally, the ethical implications of creating and distributing autonomous AI agents should be carefully considered. Nonetheless, if executed properly, this concept has the potential to unlock innovative use cases and services in the realm of AI and blockchain technology.

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#105

Earlier quoted context omitted.

It does matter, because the flat earther isn't to likely make something up about everything they talk about. They can communicate their world view, and you quickly start to figure out a model of theirs as you talk to them. None of that is true with an LLM. Any subject matter (astronomy, weather, cooking, NFL games, delegate callback methods on iOS classes, restaurants, etc) at all can have complete plausible sounding…

Fair enough. Your point is valid and I hate to be that person, but.. > It does matter, because the flat earther isn't to likely make something up about everything they talk about. I am less optimistic about this. It seems to me you are vastly overestimating the average person's rationality. Rational types are overwhelming minority. It always amazes me how even my own thin layer of rationality breaks down so very fast…

If you ask a flat earther where they recommend eating, they’re not going to interweave restaurants that exist with restaurants that don’t, but have plausible sounding restaurant names. Or if you ask for the web address of those restaurants, the flat earther will say “I don’t know, google it.” They won’t just make up plausible sounding URLs that don’t actually exist.

Hallucinations for LLMs are at a different level and approach every subject matter. Because it’s all just “predict the next word,” not “predict the next word but only if it makes sense to do so, and if it doesn’t, say you’re not sure.”

Re: Introducing Agents in Haystack: Make LLMs resolve complex tasks

#106

Earlier quoted context omitted.

Fair enough. Your point is valid and I hate to be that person, but.. > It does matter, because the flat earther isn't to likely make something up about everything they talk about. I am less optimistic about this. It seems to me you are vastly overestimating the average person's rationality. Rational types are overwhelming minority. It always amazes me how even my own thin layer of rationality breaks down so very fast…

If you ask a flat earther where they recommend eating, they’re not going to interweave restaurants that exist with restaurants that don’t, but have plausible sounding restaurant names. Or if you ask for the web address of those restaurants, the flat earther will say “I don’t know, google it.” They won’t just make up plausible sounding URLs that don’t actually exist. Hallucinations for LLMs are at a different level an…

I understand, it’s a failure mode unique to LLM’s. What I mean is that it has no relation with intelligence. Humans have failure modes too and often quite weird an surprising ones too, but they are different. It’s just that we biased and used to it.
Post reply on HN