Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UFO: A UI-Focused AI Agent for Windows OS Interaction

github.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#41

Post by nicklecompte »

g3ol4d0 wrote: post #32
DonHopkins wrote: post #31

I had ChatGPT analyze some Factorio screen dumps, which was interesting: https://docs.google.com/document/d/14p_iPhIKjDoTGa2Zr_5gPV9_...

is it somewhat right? I don't know much about factorio

Looking at it myself it's somewhat right, but it's an enormous pile of useless LLM blather - the response to "analyze the factory layout and activity" has nothing actionable, and most comments are generic ("well-organized and appears to be optimized for the efficient production of science packs"). Likewise the "analyze and optimize" requests don't seem useful enough for players to do anything with, though a lot of that is the prompt. As with most LLM stuff it just seems like someone threw a bunch of Factorio forum/Reddit posts in a blender and filtered out the stuff that was blatantly wrong.

Some specific errors I did notice:

- In the section with a processor unit subfactory, "The specific items being manufactured are not directly visible because the UI for the assembling machines is not expanded to show their recipes" is false, the player has pressed 'Alt' and the items being manufactured are shown. So this part of the response is plain wrong.

- " There are four distinct colors of science packs visible on the conveyor belts: red, green, blue, and purple, corresponding to the various levels of research complexity in the game." Only red/green/blue are shown. ChatGPT didn't make additional references to purple science but it was odd that it mentioned them at all.

- "In this Factorio image, we see a railway intersection that includes train signaling and a train crash" this is a deadlock, nothing has actually crashed. This is a minor nitpick but ChatGPT repeats the error throughout the analysis. And it also suggests that ChatGPT might not fully grok the "race condition" side of train scheduling, since deadlocks occur specifically because you're trying to avoid collisions.

I didn't want to spend too much brain calories reading all 20 pages in depth :) My general conclusion is that it's not useful enough for GPT-assisted Factorio play, and too flaky for any sort of automation of trivial Factorio tasks. I think it's plausible to make a FactorioGPT, but I doubt OpenAI's pretraining and RLHF resources covered this specific niche.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#44

Post by batch12 »

cuckatoo wrote: post #42

They can't migrate control panel to the settings app in a time measured in decades, but I'm supposed to believe that they're going to magically produce something like this that does what it claims? Go home Microsoft. You're drunk.

It sounds like the software didn't work out when you tried it. What issues did you have? Or were you using this chance to dunk on M$ without actually giving it a shot? I think Microsoft has its issues, but this response was boring.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#45

Post by smashah »

tiborsaas wrote: post #33
smashah wrote: post #22

If anyone else did this then some useless lawyer would be activated from their slumber to send out OSS devs cease & desists based on supposed violations of sacred terms and conditions. This project is the future that we were promised but it is under threat by supposed legal challenges. What if the demo asked the email to be sent as a message via whatsapp desktop instead? That would (according to their anti-freedom la…

I don't understand why anything in the demo would count as T&C violation. Who would be offended on what basis? If I can automate my desktop with a preprogrammed script, then this is not different.

As I said, if the demo opened Whatsapp and sent a message then their lawyers would harass the OSS developer of the tool with C&Ds and they'd consider it a breach of their ToS and block your account.

This is not hyperbole, that's what they will do based on precedent. And it doesn't matter about the validity of their claims because the calculation as a victim to these legal threats is that this $800bn megacorp is able to ruin your life for what amounts to less than pocket change and these big law firms are incentivized to come after you.

Adversarial interop should be an inalienable digital human right. This way companies will be forced to give API access or risk interoperability being legally scraped against their will.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#46

Post by smashah »

klabb3 wrote: post #34
smashah wrote: post #22

If anyone else did this then some useless lawyer would be activated from their slumber to send out OSS devs cease & desists based on supposed violations of sacred terms and conditions. This project is the future that we were promised but it is under threat by supposed legal challenges. What if the demo asked the email to be sent as a message via whatsapp desktop instead? That would (according to their anti-freedom la…

Rules for thee not for me. Scraping for instance is bad only when other people do it. Technically speaking though, user facing websites are built with quite weak UI accessibility in mind. Partly to prevent others bots from using it. I worry less about the LLM/AI particularities, and more about adding yet another comprehensive layer to the stack. It’d not exactly be standing on solid foundation. During the 2010s Web 2…

The promises of web 2.0 were stolen from us.

Adversarial interop should be a digital human right.

Put an API layer in front of UFO and all of a sudden we're one step closer to the unshittification of our digital lives.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#47

Post by kQq9oHeAz6wLLS »

DonHopkins wrote: post #24

Anyone who uses the term "work smarter not harder" in a demo video, or at any point in their life in any situation, should not be taken seriously.

That rules out every parent trying to teach their kids to think critically. I don't think you'll like where that leads us.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#48

Post by austinwade »

cuckatoo wrote: post #42

They can't migrate control panel to the settings app in a time measured in decades, but I'm supposed to believe that they're going to magically produce something like this that does what it claims? Go home Microsoft. You're drunk.

The engineers who built this UFO project probably would have been able to move Control Panel to settings. Microsoft is a big company.

Re: UFO: A UI-Focused AI Agent for Windows OS Interaction

#50

Post by MaxikCZ »

batch12 wrote: post #44
cuckatoo wrote: post #42

They can't migrate control panel to the settings app in a time measured in decades, but I'm supposed to believe that they're going to magically produce something like this that does what it claims? Go home Microsoft. You're drunk.

It sounds like the software didn't work out when you tried it. What issues did you have? Or were you using this chance to dunk on M$ without actually giving it a shot? I think Microsoft has its issues, but this response was boring.

I like seeing the context, imo it is important.
Post reply on HN