Does it support other visual-input-accepting language models? GPT-V is paywalled.
The code isn’t very complicated. You could edit in any vision model you want: https://github.com/microsoft/UFO/blob/main/ufo/llm/llm_call....
UFO: A UI-Focused AI Agent for Windows OS Interaction
51–60 of 66 posts
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#52Note this is from Microsoft! They are on a roll lately, and seem to have beaten OpenAI to GPT-Agents with this release.
Hard to "beat" someone when you have a very close partnership and you're using their model (GPT-4V), likely this is developed quickly with input from OpenAI. I don't think it's right to think of the two as competing at this point.
OpenAI gave Microsoft the model weights, and Microsoft hosts it on Azure for MS Research. None of the Azure usage analytics goes back to OpenAI except via bug reports and publications.
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#53Earlier quoted context omitted.
It sounds like the software didn't work out when you tried it. What issues did you have? Or were you using this chance to dunk on M$ without actually giving it a shot? I think Microsoft has its issues, but this response was boring.
I like seeing the context, imo it is important.
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#54Sometimes these are automated with ”robotic procesd automation” tools. Something like UFO could streamline the process.
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#55On the other hand, I shudder to think of the millions of man hours required to arrive at this solution, when simple UI guidelines, or better yet, an API, would have solved my problem far more simply and efficiently.
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#56Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#57Earlier quoted context omitted.
I like seeing the context, imo it is important.
How is not moving the control panel important context or related to this product? I'm not seeing it.
When a game studio releases new game, looking at previous games they released is a solid indicator of what quality we can expect. Same goes for software companies making new piece of software. Combined with how LLMs seem super useful until I want them to do something original, I would be very sceptical even in case a corp like Apple came up with this announcement. Seeing it comes from Microsoft, not trusting it isn't result of "hating them is popular", its result of experience.
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#58have they fixed teams yet? that app that like the entire united states uses for pc work every day? i still cant copy paste a code block, or copy paste literally anything. i think microsoft should use AI to learn how to code code blocks in chat or they should ask chatgpt how to use the clipboard of their own OS
Yes of course. It works perfectly for what it was meant to be: good enough so people won't switch away from MS ecosystem to have a group chat. Nothing more. Teams is actively developed along the lines that deliver greatest value to Microsoft at the expense of their customers (a business model increasingly popular these days): 1) implementing corporate-nerfed versions of vanity features introduced by competitors in gr…
Re: UFO: A UI-Focused AI Agent for Windows OS Interaction
#59Earlier quoted context omitted.
Rules for thee not for me. Scraping for instance is bad only when other people do it. Technically speaking though, user facing websites are built with quite weak UI accessibility in mind. Partly to prevent others bots from using it. I worry less about the LLM/AI particularities, and more about adding yet another comprehensive layer to the stack. It’d not exactly be standing on solid foundation. During the 2010s Web 2…
The promises of web 2.0 were stolen from us. Adversarial interop should be a digital human right. Put an API layer in front of UFO and all of a sudden we're one step closer to the unshittification of our digital lives.