Live data from Hacker News

Self Driving Desktop

github.com

11–20 of 39 posts

Re: Self Driving Desktop

#11
post #6
post #4

Earlier quoted context omitted.

I did have a recording function around, to track mouse movement. The issue is that the mouse movement gets verbose, and you would have to clean that up somehow.

Sounds like a candidate for machine learning - and an excuse to learn it.

nah, just xlib events being printed to tty

Re: Self Driving Desktop

#12
What is it? The page just says "Desktop Automation framework" and then lists a bunch of commands and switches.

Perhaps 2-3 paragraphs describing what it does?

Re: Self Driving Desktop

#14
One of the most unknown tools (>10 years) http://sikulix.com/ It allows play mouse/keyboard event scrips BUT it allows to find components (coords) via screen OCR so you can make your scripts multi resolution/desktop independent. Also, it's Java based so you can play it multi SO.

Re: Self Driving Desktop

#16
post #14

One of the most unknown tools (>10 years) http://sikulix.com/ It allows play mouse/keyboard event scrips BUT it allows to find components (coords) via screen OCR so you can make your scripts multi resolution/desktop independent. Also, it's Java based so you can play it multi SO.

This is an excellent tool! But you forgot to mention that the user codes it in python, it comes with a purpose built ide, it recognises both text and images the latter with an approximation capability.

Re: Self Driving Desktop

#17
> mv x y s;: move the mose to x,y in s seconds

The problem with tools like this is that they create an API that the developers don't know about and have no intention of supporting. I broke one recently by having the app maximize on startup, but everything from adding UI elements, rearranging them or timing differences can introduce breakages.

Considering it's scripting anyway, an actual API would be easier.

Re: Self Driving Desktop

#18
post #16
post #14

One of the most unknown tools (>10 years) http://sikulix.com/ It allows play mouse/keyboard event scrips BUT it allows to find components (coords) via screen OCR so you can make your scripts multi resolution/desktop independent. Also, it's Java based so you can play it multi SO.

This is an excellent tool! But you forgot to mention that the user codes it in python, it comes with a purpose built ide, it recognises both text and images the latter with an approximation capability.

There are a number of other commercial and free desktop automation tools that exist, some of which I've used to automate GUI testing in the past.

https://en.m.wikipedia.org/wiki/Comparison_of_GUI_testing_to...

My favorite on the Windows side was vTask Studio, but it looks like the domain is down and the link was removed from that wiki page.

Re: Self Driving Desktop

#19
post #4
post #3

Earlier quoted context omitted.

I use OBS for recording and Flowblade for editing. Got sick of editing my mistakes out, so then this repo came to be. Planning to add some playlists to start that up, set file names, begin/end recording. self-driving-desktop will be part of a demo automation framework that is in the progress.

I did have a recording function around, to track mouse movement. The issue is that the mouse movement gets verbose, and you would have to clean that up somehow.

if you could access window dimensions and if a mouse click yielded an action, you could probably back out that click’s coordinates to that button and toss the rest of the cursor data.

Re: Self Driving Desktop

#20
post #14

One of the most unknown tools (>10 years) http://sikulix.com/ It allows play mouse/keyboard event scrips BUT it allows to find components (coords) via screen OCR so you can make your scripts multi resolution/desktop independent. Also, it's Java based so you can play it multi SO.

Sikuli is amazing! I've used it (to great success) for data processing automation and MMO grinding.
Post reply on HN