The amount of hype around AutoGPT is probably good for other people building similar things though. It allows people to build without all the grifter attention that AutoGPT gets.
AI Report #4: AutoGPT And Open-source lags behind Part 2
11–20 of 37 posts
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#12Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
The search plugin is bad. I have done some exploratory work [1] on specializing search agents to understand the query and result syntax and use that to structure results but I started with an explicitly structured data source to prove the concept and then kinda wandered off.
It was my hypothesis that the variety of trash returned by things like serpapi need to be massaged into something consistent and potentially run through a result retrieval and fine tuning stage to be useful to a high level agent like autogpt, but didn't make it far enough to have anything working to show.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#13I don't like how this newsletter uses "we/our" terminology - "Our personal take is that it might be beneficial to build agents that excel at some really well-defined and small-scale tasks" - without (as far as I can see) ever saying who "we" are.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#14Earlier quoted context omitted.
Were you using it with gpt3 or 4? Personally, I can’t get gpt4 api access despite being a PI at a well known research institution with a project that would be great publicity for OpenAI. My theory is that people are running it mostly with 3 and then saying it’s useless. It definitely is useless with 3.
Are you an OpenAI user who has a valid payment method attached to your account? I've notice they approve GPT-4 as long as you have a valid payment method and a reasonable justification, which in my case one liner.
There's a waiting list to use the GPT-4 API. https://openai.com/waitlist/gpt-4-api
Anecdotally (and unsurprisingly), they seem to be prioritising those with "value-add" use cases in a variety of industries over individuals just wanting to play.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#15Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
That about matches my experience. It's a neat project but that's about it.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#16Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
The ratio of 45 second “Twitter video demos” vs. examples of actual code/prompts/real world use cases you can replicate is quite striking. Dipping into related discords, I feel like I’m always missing something obvious because there is so much activity but what feels like to me so little replicable substance. I’m a terrible coder so I partially chalk it up to that but it definitely seems like it’s hitting the current boundaries of a parrot echoing itself into gibberish.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#17Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
Can you help me understand autogpt? Is it just a recursive gpt where an initial prompt is given and has the ability of the output to be used to pipe to other gpt prompts. Am I missing something? I tried doing more complex tasks using GPT4 and was initially optimistic about plugins but they have all been very disappointing. For instance, a dream for me would be something like: "Find some rental property opportunities…
This is exactly the promise and there’s a number of handwavey demos, and it feels like it should be easy enough to have something exactly like this as a “hello world,” but I haven’t seen any of the auto GPT-type that can reliably execute even a basic version of this. As others have mentioned, a little scaffolding custom for the project can work great, but having GPT build that scaffolding isn’t there, as far as I can see (I think a lot of people could benefit from a step-by-step of the parent’s use case as a proof of concept).
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#18Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
Yeah I played around with it for a few days to improve my python code, and narrowed the command space enough for it to be helpful 50% of the time
here's the video: https://www.loom.com/share/5e83475be2464778950f7df7e209ac2d
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#19I know a big part of the discussion around this link is around whether AutoGPT is viable or not, but I think a more interesting piece are the papers linked at the bottom (especially the "Tool Maker" one). This pattern of "well, maybe the LLM can do that too" is just so obvious but so meta and simultaneously brittle. "LLM can use tools" -> "The tools can, themselves, be LLM-based" -> "The LLM can create it's own tools that it then uses".
It's really turtles all the way down.
Re: AI Report #4: AutoGPT And Open-source lags behind Part 2
#20Has anyone actually done anything of use with autogpt? I tied it for 4 tasks and it would inevitably get stuck on each and produce absolutely nothing of value. These were fairly simple like research popular topics and write articles on them etc. It would do things like Google something, the result wasn’t relevant , try again , got an error from one of the pages and then seemingly started to do something completely in…
It is likely we can do better than 1:1 human input to GPT output on current tech; but the human in the loop is doing a lot of work very easily that the LLM is very bad at, just like the LLM is doing a lot of work very easily that is otherwise laborious for the human. We can't just take the things that the LLM is bad at and humans do easily and expect to fix it with more LLM.
Right now we have:
Step 1: Human reasoning, tool use, input.
Step 2: LLM output.
Step 3: Human reasoning, tool use, input.
Step 4: LLM output.
&etc.
The observation that the input and output are both just text makes it possible to make "agents". But the "agent" movement trying to totally close the whole loop right away is way too early.
It's fine to lay the groundwork though, and the frameworks for it, like AutoGPT, can be used to just do a couple extra steps rather than close the whole loop.
Plugins and browsing can be seen as merging some of step 2 and 3. But then you still need the &etc iteration with the human closely in the loop.
Chain of thought prompting techniques are similarly an attempt to merge a little bit of the human's process of vetting the output by trying to get better output in individual iterations. Sometimes I make the LLM output multiple options and pick the best one with its reasoning; this is really just compressing multiple runs of the LLM and having it pick one, rather than me retrying if I get a bad output.
Anyway I think this is the right way to look at it; these are good tools for trying to compress iterations of human-in-the-loop. For some things maybe we'll eventually remove the human, but we shouldn't expect it right now. The twitter demonstrations of "it did the whole thing" are a trick; good for influences, but not realistic right now.