Access to technology is a key part of living an independent life. And for those living with high quadriplegia from a spinal cord injury (tens of thousands in the U.S. alone), this usually means finding a different strategy to access technology than the touchscreens and keyboards that most devices seem to rely on at first glance.
Fortunately, there have been several recent developments in touch-free access, on inexpensive and widely available hardware. For mobile devices, in particular, there are now options for total device control based on either head tracking or voice, for both Android and iOS.
Head Tracking
For supported devices on Android, our team built Jabberwocky Touch-Free Accessibility, a free app to control your entire device based on head movement. You have access to email, games, internet, video, and all existing Android apps. It uses the device’s front-facing camera and proprietary artificial intelligence technology to track head movement and display a cursor where your nose is pointing on the screen. Open and close your mouth to perform a tap at the location of the cursor. To swipe or perform complex real-time gestures, open your mouth and drag the cursor across the screen with head movement, and then close your mouth to end the gesture.
Jabberwocky Touch-Free Accessibility is currently available for free on the Android platform, for any device that runs Android 8.0 or higher. We chose the Android platform for our total accessibility app because Google embraces third-party assistive technology developers by allowing them to build Accessibility Services, apps with elevated permissions necessary to control the entire device.
Apple, by contrast, has a restrictive security model that prevents third-party app assistive technology developers from building an app that controls the entire device. Unfortunately, this policy, which attempts to restrict the reach of malicious developers, dramatically reduces the capability of teams outside of Apple to contribute to their accessibility solutions. For example, we were successful in building a head-tracking-based web browser, which enables email, shopping, YouTube, etc., but not an app that controls the entire device.
As of iOS 13, Apple does offer its head-tracking accessibility service on devices with a TrueDepth camera (iPhone X and later, iPad Pro 2018 series). For more information, head to this webpage and search for “head tracking”. There are some caveats. For example, using a facial gesture such as a smile to click on the screen causes the cursor to jump around, resulting in misclicks. Try it out for yourself and see what you think.
Voice Control
For controlling your device with your voice, there have been dramatic improvements even within the last year or so. Both Android and iOS now offer a first-party solution: Google Voice Access for Android and Voice Control for iOS. These go well beyond the now-limited capabilities of something like Siri in terms of unlocking the whole device. You can click on elements of the screen, scroll, dictate, and use every aspect of the phone with your voice.
What's best for me?
Ultimately you will decide what’s best for you, ideally after trying out the options. In general, the “pros” for head tracking are:
- simplicity,
- finer-grained control and efficiency, and
- better suitability for games and browsing.
The biggest “cons” for head tracking are:
- it requires the use of the neck, sometimes more than a user might have available, and
- It is “always-on” (though usually, there is a simple gesture or action to disable/enable)
Some “pros” for voice control are:
- speaking is already a learned physical action,
- basic commands like starting an app are super-easy, and
- it has great integration with speech-to-text.
“Cons” for voice control are:
- its inefficiency for some tasks (like clicking on a specific part of the screen),
- the high learning curve on the different voice commands, and
- privacy in some cases (if speaking in public).
Both head tracking and voice control can behave worse in less-than-ideal environments (bright lighting for head tracking and ambient noise for voice control). But it is surprising to see how much both are becoming resilient to distracting signals.
At the end of the day, find what works for you. One of my friends became very effective at playing Words With Friends on his phone using a stick attached to his forearm. No solution is perfect, but hopefully, now you have a few more options to check out!
Learn more about Jabberwocky for Android:
- Download Jabberwocky free on Google Play: https://play.google.com/store/apps/details?id=org.swiftable.jabberwocky
- Jabberwocky 2.0 for Android press release: https://www.jabberwockyapp.com/blog/2020/10/26/jabberwocky-20-released
