Text editing on mobile: the invisible problem

It's kind of sad to reflect that the Apple Newton had this almost a decade prior[1] to the first iPhone and in an intuitive manner, by selecting text by highlighting it with the stylus then dragging it to the edge of the screen where it would remain in a truncated pill form for later retrieval (dragging out the pill to a document). No menus involved at all.

[1] @ 30 seconds in: https://www.youtube.com/watch?v=Sue2BR1AHUE

No, it's completely broken. And mobile is an absolute mess. The touch system is so utterly overloaded with functionality that it effectively makes all these wonderful things completely opaque and, in a lot of cases, not discoverable at all.

Ask any iOS user what single, double, triple clicks mean. Ask them if they know about long touch on the spacebar. Ask them to show you everything they know about how to operate the device.

What you'll learn is that the vast majority of people pretty know nothing about the function overloading of touch and gestures. They know simple touch interaction and that's about it.

The touch screen, whether it is on a phone or in a car, exists for one simple reason: It is a lot easier to make UI decisions when you don't have to make them. It's cheaper. It's faster to design. Manufacturing costs are lower. Testing is simpler.

Having design many physical control panels during my career, for products ranging from commercial through aerospace, I can tell you it is a difficult grueling process. In short, pain in the ass. And, if you make a mistake or change your mind, you can't just compile your way out of it. Hardware is hard (and expensive) for a reason.

Today's smart phone designers flat-out refuse to add physical UI elements to their phones, instead, choosing contorted ways to overload the touch interface.

Prior to iOS devices I owned some very interesting phones and PDA's by companies like Motorola, Blackberry and Palm. In the aggregate, these companies had interesting physical UI ideas that worked very well.

A stylus, for example, is brilliantly simple and better than your finger, particularly if more sophisticated, as seen on Wacom tablets.

Blackberry had a small trackball you could also click right on the keyboard (another very useful UI element). It was great for editing text and other use cases. Heck, you could use a browser with a proper desktop-style cursor.

Motorola had a click-wheel on a PDA-like add-on to their flip phones. Fantastic for navigation and super easy to use. The clip-on also had additional buttons, that made such things as single-handed almost-blind address book navigation very simple.

Also, touch screens are incredibly fragile. Example, during a call, any accidental touch will cause problems. I have resorted to switching away from the phone application during my call and, if I have to move around (while wearing a headset) placing the phone facing outward in my pant pocket. I've had too many cases of the phone doing stupid things during a call when in my pocket facing inward.

In short, touch screens are terrible for the consumer in a lot of use cases. They are fantastic for the manufacturer because they don't have to make any decisions, manufacture a single product and all is well so long as the torture they inflict on their customers is just below a pain threshold.

There are so many potentially interesting ideas that could be explored. Adding a set of user-programmable physical macro buttons somewhere (sides or back). Adding a small cursor-only touch pad to the back for, well, pointing control. Click and right click could be actual buttons. Blackberry's mini trackball was brilliant. And a real keyboard, as a first-class citizen add-on, not something left to third parties with no real support past the basics.

If someone wants to make heavy use of a device they would love a real full-width keyboard that clips-on and folds open with a Blackberry style trackball and a few extra keys and functions. Lots of people buy cases for their phones. I would absolutely buy an official fold-out keyboard/case if it was truly integrated so I could actually use the thing without having to remember what triple-click-press-slide-right-and-face-north f-ing means.

For me, the phone is unusable for anything beyond the trivial. I hate browsing the internet with it. I don't buy anything other than Uber rides with it. I don't use it to buy anything on Amazon. Text messaging for work is useless other than the very basic, because it is horrible to edit anything and there is no real desktop integration to track and search as you can with an email client.

Which brings me to the other problem with mobile:

It's 2023.

Why isn't my phone seamlessly integrated with my computer?

When I plug-in or otherwise connect my phone to my desktop or laptop, the phone's screen should appear on my desktop as a first-class citizen for full interaction without having to touch my phone. I should be able to edit text on my phone from my desktop in a phone window. I should be able to cut, copy and paste anything between desktop and mobile. I should be able to run applications on my phone and interact with them on one of my three 27 inch screens. I should be able to click on a phone number on the desktop and dial it on the phone. I should be able to use the same headphones I use on my desktop to run a call on the phone through the desktop interface. I should be able to move, copy and manage files on the phone from my laptop. Etc.

I don't really understand what mobile designers are doing. It seems that they stopped being creative and holding their user's best interests as their primary guidance. They slap a touch screen on an aluminum frame, use as few buttons as absolutely possible. Limit desktop integration to the absolute minimum (iTunes is garbage) and leave third parties to hack stuff together rather than develop exquisitely integrated solution.

I could be wrong on this. I firmly believe Microsoft still has a real shot at a next generation phone. Their Windows Phone attempt was a failure for many reasons. What is lacking today are the things I mentioned above, and more. As ridiculous as it may sound in 2023 --because it should have been done by now-- having a device that fully integrates with desktop in an exquisitely usable symbiotic relationship would be nothing less than amazing. That, I believe, would gain traction.

The problems are with editing only -- precise cursor positioning, precise selection, and then access to basic cut/copy/paste/undo operations that doesn't mess everything up.

My hunch is that we need a button on mobile keyboards to switch to a kind of "edit gesture mode". Some kind of swipe area to move the cursor, some kind of swipe area/mode to extend/contract a selection, some method to handle scrolling as necessary, some kind of magnifying zoom to select tiny things like narrow punctuation, and separate larger button areas for cut/copy/paste/undo. Maybe instead of swipes there are gestures in a kind of dedicated trackpad-type area of the screen, I don't know.

But I definitely think there's a ton of area for experimentation that hasn't been explored yet. The hold-spacebar-to-turn-keyboard-into-trackpad-to-move-cursor mode was a first step the iPhone took towards this, but I think it can go 20x further.

I think it's something that only Apple and Google are capable of developing right now though. I don't think there are enough API's exposed for third-party keyboards to directly control things like text selection, zoom, scrolling, cut/copy/paste/undo, and the like.

I believe that the relation should be reversed: Make the best system possible, usable by well-trained users, and only then add easy modes, tutorials, child safe modes, etc. with paths for improvement at every level. But when using modern systems, I often feel like I’m reduced to using Fisher-Price toys with one (inoperative) button.

I have a keyboard that I've attached a guitar strap to, in order to be able to type while standing, without a table to rest the keyboard on.