Live data from Hacker News

UFO: A UI-Focused AI Agent for Windows OS Interaction

github.com

51–60 of 66 posts

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#51

Does it support other visual-input-accepting language models? GPT-V is paywalled.

The code isn’t very complicated. You could edit in any vision model you want: https://github.com/microsoft/UFO/blob/main/ufo/llm/llm_call....

I don't have the resources currently but would love to test something like LLaVA out for this, but I won't keep my hopes up.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#52
post #13
post #5

Note this is from Microsoft! They are on a roll lately, and seem to have beaten OpenAI to GPT-Agents with this release.

Hard to "beat" someone when you have a very close partnership and you're using their model (GPT-4V), likely this is developed quickly with input from OpenAI. I don't think it's right to think of the two as competing at this point.

If I understand correctly, there is zero overlap between Microsoft AI Research efforts that use GPT4(V) and OpenAI's development.

OpenAI gave Microsoft the model weights, and Microsoft hosts it on Azure for MS Research. None of the Azure usage analytics goes back to OpenAI except via bug reports and publications.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#53
post #50
post #44

Earlier quoted context omitted.

It sounds like the software didn't work out when you tried it. What issues did you have? Or were you using this chance to dunk on M$ without actually giving it a shot? I think Microsoft has its issues, but this response was boring.

I like seeing the context, imo it is important.

How is not moving the control panel important context or related to this product? I'm not seeing it.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#55
On one hand, I'm really glad to have come across this this morning, as I'm trying to automate a task that requires GUI automation due to a lack of an API. But I'm stuck even on that because some windows don't accept key input to select the "Enter" button (why? no idea), so now I'm having to drop into mouse automation, which of course is super imprecise, being DPI-dependent. So, taking screenshots and then feeding those into a LLM is a solid solution, I guess.

On the other hand, I shudder to think of the millions of man hours required to arrive at this solution, when simple UI guidelines, or better yet, an API, would have solved my problem far more simply and efficiently.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#57
post #53
post #50

Earlier quoted context omitted.

I like seeing the context, imo it is important.

How is not moving the control panel important context or related to this product? I'm not seeing it.

Context lies in how this corporation approaches bars it set itself, and UX the end user have to deal with resulting from it.

When a game studio releases new game, looking at previous games they released is a solid indicator of what quality we can expect. Same goes for software companies making new piece of software. Combined with how LLMs seem super useful until I want them to do something original, I would be very sceptical even in case a corp like Apple came up with this announcement. Seeing it comes from Microsoft, not trusting it isn't result of "hating them is popular", its result of experience.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#58
post #10

have they fixed teams yet? that app that like the entire united states uses for pc work every day? i still cant copy paste a code block, or copy paste literally anything. i think microsoft should use AI to learn how to code code blocks in chat or they should ask chatgpt how to use the clipboard of their own OS

Yes of course. It works perfectly for what it was meant to be: good enough so people won't switch away from MS ecosystem to have a group chat. Nothing more. Teams is actively developed along the lines that deliver greatest value to Microsoft at the expense of their customers (a business model increasingly popular these days): 1) implementing corporate-nerfed versions of vanity features introduced by competitors in gr…

Copy paste? What are you trying to do, exfiltrate company secrets? Security turned off that feature to protect us all.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#59
post #46
post #34

Earlier quoted context omitted.

Rules for thee not for me. Scraping for instance is bad only when other people do it. Technically speaking though, user facing websites are built with quite weak UI accessibility in mind. Partly to prevent others bots from using it. I worry less about the LLM/AI particularities, and more about adding yet another comprehensive layer to the stack. It’d not exactly be standing on solid foundation. During the 2010s Web 2…

The promises of web 2.0 were stolen from us. Adversarial interop should be a digital human right. Put an API layer in front of UFO and all of a sudden we're one step closer to the unshittification of our digital lives.

Just wait until application designers start to create machine-hostile interfaces to prevent vision models from recognizing what to click, or introducing rate limits on UI interactions. There is a strong desire to control how one's product (app, website, or whatever) is used.
Post reply on HN