Professors Staffed a Fake Company with AI Agents, Guess What Happened?
1–10 of 21 posts
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#2[deleted]
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#3Clickbait headline, and it's reporting something from Business Insider (itself IMO a terrible website these days), but:
> the results were dismal. The best-performing model was Anthropic's Claude 3.5 Sonnet, which struggled to finish just 24 percent of the jobs assigned to it. The study's authors note that even this meager performance is prohibitively expensive, averaging nearly 30 steps and a cost of over $6 per task.
and other AIs were worse.
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#4It ended humanity's existence? No?
Not yet? Okay. Good. In fact, great! I like existing.
For now.
"Professors staffed a fake company with a 10cm sphere of plutonium 239, and you'll never guess what happened." Egg on their face, I'm sure.
Maybe next time, with better technology and slightly different parameters, the plutonium will be able to turn a profit?
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#5> is arguably still just an elaborate extension of your phone's predictive text
Nailed it. It seems to be doing a good job of helping coders and document writers. It seems to be great at solving protein folding. Other than that, I'm not so sure.
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#6[deleted]
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#7Clickbait headline, and it's reporting something from Business Insider (itself IMO a terrible website these days), but: > the results were dismal. The best-performing model was Anthropic's Claude 3.5 Sonnet, which struggled to finish just 24 percent of the jobs assigned to it. The study's authors note that even this meager performance is prohibitively expensive, averaging nearly 30 steps and a cost of over $6 per tas…
$6 per task does not sound prohibitively expensive to me, quite the opposite.
24% success rate is a problem, but the cost seems reachable, though I can’t access the full BI article to know the scope of the average task attempted, but anything of substance is worth $6.
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#8CMU professors can't build AI agents, and decide to brag about it. That's the article.
"We tried something, and we couldn't make it work. Therefore it must be impossible to do."
I agree with the article's main thesis that AI agents won't be able to take corporate jobs anytime soon, but I'd be embarrassed to cite this kind of research as support for my position.
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#9Has anyone figured out how to hook up LLMs to Mechanical Turk, and have revenues greater than expenses? Or is this akin to the net energy problem in fusion?
Re: Professors Staffed a Fake Company with AI Agents, Guess What Happened?
#10CMU professors can't build AI agents, and decide to brag about it. That's the article. "We tried something, and we couldn't make it work. Therefore it must be impossible to do." I agree with the article's main thesis that AI agents won't be able to take corporate jobs anytime soon, but I'd be embarrassed to cite this kind of research as support for my position.
It’s not entirely clear from the write up in the article, but it sounds like this was intended as a test of existing “off the shelf” AI agent models. In other words, the aim is to find out what happens if you try to use the existing commercially available technology (which of course is what most people would be doing).