Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Warpd: A modal keyboard-driven virtual pointer

github.com

Post reply on HN
Print view View original

21–30 of 125 posts

Re: Warpd: A modal keyboard-driven virtual pointer

#21

Post by hawski »

_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

Very interesting thing. I wonder if the gap of apps not supporting a11y could be reduced by using Tesseract to OCR the text.

Re: Warpd: A modal keyboard-driven virtual pointer

#22

Post by discodachshund »

_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

Excited to try this out! Is it planned to open source? I would love to try integrating this into Raycast

Re: Warpd: A modal keyboard-driven virtual pointer

#23

Post by epse »

awestroke wrote: post #10

Looking at the limitations, I hate how fragmented Linux is becoming for apps like this. Completely separate implementations for: - X - Wayland + Gnome Shell - Wayland + wlroots-based compositor - Wayland + sway or any other non-wlroots based compositor Is Wayland really that much better that this is worth it? Why can't Wayland be aware of the compositor like X?

Wait is there a separate implementation for sway and other wlroots? Because sway is wlroots based, that's where wlroots originates?

Re: Warpd: A modal keyboard-driven virtual pointer

#24

Post by _chendo_ »

kache_ wrote: post #20
_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

Have you considered using ML/OCR to figure out the position of the text relative to the screen? Seems much simpler than relying on accessibility APIs Thank you for your hard work!

I have plans to use ML/OCR to augment results down the road but the AX APIs and ecosystem on most apps (that I encounter, at least) are generally decent. Also, OCR means it won’t understand buttons with just icons, whereas AX APIs can grab em just fine.

Thanks! It’s easily my longest running project at a decade

Re: Warpd: A modal keyboard-driven virtual pointer

#25

Post by punnerud »

discodachshund wrote: post #22
_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

Excited to try this out! Is it planned to open source? I would love to try integrating this into Raycast

Is Raycast open source? Could only find that the plugins are on GitHub.

Re: Warpd: A modal keyboard-driven virtual pointer

#26

Post by Luc »

_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

This app is quality. I can tell you have been working on it for years. Why not charge for it?

Re: Warpd: A modal keyboard-driven virtual pointer

#27

Post by vidarh »

awestroke wrote: post #10

Looking at the limitations, I hate how fragmented Linux is becoming for apps like this. Completely separate implementations for: - X - Wayland + Gnome Shell - Wayland + wlroots-based compositor - Wayland + sway or any other non-wlroots based compositor Is Wayland really that much better that this is worth it? Why can't Wayland be aware of the compositor like X?

Personally I don't think Wayland will get to a good spot until one compositor "wins" and provides interfaces for modification/extension such as e.g. X-like ability to write window managers without replacing the whole compositor. At this point I see no compelling reason to switch to Wayland for my own part.

Re: Warpd: A modal keyboard-driven virtual pointer

#28

Post by yewenjie »

_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

I have been meaning to build something like this for myself, albeit for Linux. Does anybody know if there is any already existing efforts there?

Given that Linux doesn't have anything like an accessibility API, I think the only option is training ML models.

Re: Warpd: A modal keyboard-driven virtual pointer

#29

Post by xcambar »

pomatic wrote: post #16
hunogo wrote: post #12

Earlier quoted context omitted.

Weird.

Some people need tools like this, as an assistive technology - think RSI, Parkinsons and other issues that affect dexterity or elbow movement. Not so weird.

Thank you for the examples of necessity over examples of preference.

Re: Warpd: A modal keyboard-driven virtual pointer

#30

Post by pabs3 »

_chendo_ wrote: post #9

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [ https://shortcat.app/ ] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on…

Should be doable on Linux for most mainstream apps due to the toolkits having a11y support, but obviously not all apps use mainstream toolkits.
Post reply on HN