Notes on Vision Pro

Eye tracking is almost certainly more accurate and faster. To the point where we make it an entire game to see who can who can move the mouse to what their eyes are already looking at fastest, in the form of shooter games.

Brave's reader mode wasn't shown as an option, though my BeeLine Reader extension was able to parse the text when I invoked its Clean Mode. I was a little surprised, since the BeeLine extension didn't detect/color the text inline, like it usually does.

> But it does put an enormous amount of pressure on the eye tracking. As far as I can tell so far, the role of precise 2D control has been shifted to the eyes.

I've been researching eye tracking for my own project for the past year. I have a Tobii eye tracker which is probably the best eye tracking device for consumers currently (or the only one really). It's much more accurate than trying to repurpose a webcam.

The problem with eye tracking in general is what's called the "midas touch" problem. Everything you look at is potentially a target. If you were to simply connect your mouse pointer to your gaze, for example, any sort of hover effect on a web page would be activated simply by glancing at it. [1]

Additionally, our eyes are constantly making small movements call saccades [2]. If you track eye movement perfectly, the target will wobble all over the screen like mad. The ways to alleviate this are by expanding the target visually so that the small movements are contained within a "bubble" or by delaying the targeting slightly so the movements can be smoothed out. But this naturally causes inaccuracy and latency. [3] Even then, you can easily get a headache from the effort of trying to fixate your eyes on a small target (trust me). Though Apple is making an effort to predict eye movements to give the user the impression of lower latency and improve accuracy, it's an imperfect solution. Simply put, gazing as an interface will always suffer from latency and unnatural physical effort. Until computers can read our minds, that isn't going to change.

Apple decided to incorporate desktop and mobile apps into the device, so it seems this was really their only choice, as they need the equivalent of a pointer or finger to activate on-screen elements. They could do this with hand tracking, but then there's the issue of accuracy as well as clicking, tapping, dragging or swiping - plus the effort of holding your arms up for extended periods.

I think it's odd that they decided that voice should not be part of the UI. My preference would be hand tracking a virtual mouse/trackpad (smaller and more familiar movements) plus a simple, "tap" or "swipe" spoken aloud, with the current system for "quiet" operation. But Apple is Apple, and they insist on one way to do things. They have a video of using Safari and it doesn't look particularly efficient/practical to me [4].

But who knows - I haven't tried it yet, maybe Apple's engineers nailed it. I have my doubts.

1. https://uxdesign.cc/the-midas-touch-effect-the-most-unknown-...

2. https://en.m.wikipedia.org/wiki/Saccade

3. https://help.tobii.com/hc/en-us/articles/210245345-How-to-se...

4. https://developer.apple.com/videos/play/wwdc2023/10279/

Interesting point — if you no longer need to use a joystick to aim your weapon, how will controllers evolve? Will the second joystick be used for some other function, or will it be replaced by a different type of input method?

It would be funny if controllers evolved to be more like the single-joystick models that we had decades ago, with the joystick on the left and rows of buttons on the right. History doesn't repeat itself, but perhaps it'll rhyme?

Interesting notes.

I'm disappointed even though it's entirely predictable that VisionOS is built on the iOS/iPadOS foundation rather than OSX. I guess we'll see how "walled in" it is but it's hard to see any reason Apple isn't going to be just as constraining and controlling about what happens on this OS as they are on iOS, if not more so. Which ultimately means I'll be very reluctant to ever adopt it in any meaningful way as my primary computing device.