Goodbye to Keyboard and Mouse: How Virtual and Augmented Reality Are Rewriting the Rules of Human-Computer Interaction

By David Carneros, predoctoral researcher in training

I'm sure you've all seen videos on Twitter (or "X") of people walking down the street with Apple's new augmented/virtual reality glasses, moving their hands in a rather comical way. At first glance, these gestures may seem like a futuristic dance or a new form of secret communication (reptilian perhaps?). However, what we are witnessing is the beginning of an era in which the peripherals that we currently know could become pieces in the museum of the hall of the ESI. These glasses are redefining the way we interact with the digital world, promising a more natural and direct integration between humans and machines.

These glasses are redefining the way we interact with the digital world, promising a more natural and direct integration between humans and machines.

I remember my first contact with a computer when I was six years old, I still remember the fascination I felt when learning to use the keyboard and mouse. Little by little, these peripherals became extensions of my body, tools without which it was difficult to imagine any interaction with technology.

Over time I had my first cell phone (the legendary indestructible Nokia) and I adapted my brain to using that tiny keyboard on which to write my name I had to press 9 keys. And shortly after I experienced my first interaction revolution, touch screens appeared. Suddenly, direct contact with digital interfaces became second nature, removing a physical barrier between the user and digital content. This transition from physical to tactile was already a huge leap, but what we are witnessing with augmented and virtual reality is even more revolutionary.

The transition from physical to tactile was already a huge leap, but what we are witnessing with augmented and virtual reality is even more revolutionary.

Augmented/virtual reality glasses, such as Apple's Vision Pro or Meta's Quest, are ushering in an era in which user interfaces become even more intuitive, fluid and personalized. Interaction based on gestures and voice commands promises total immersion in the digital world, without the need for physical mediators.

Imagine for a moment our world with this new technology, in which to program we do not need a screen, a keyboard and a mouse, in which we can simply program with glasses and our hands. Well, it is something that is already possible.

Here's a video of me using Vision Pro with a 5120×2880 macOS monitor for coding and 3D design, and then other social media windows and my email open in separate windows on visionOS. You don't actually need multiple macOS monitors being mirrored imho. You can also see that my… pic.twitter.com/E3OeaxtWGi

— Geert Bevin (@gbevin) February 10, 2024

However, this transition to new forms of interaction also poses challenges. Just as I had to learn when I was little to use the mouse and keyboard, new generations will have to adapt to interfaces that may seem futuristic to us today. Additionally, considerations around accessibility and inclusion become even more critical, ensuring that no one is left behind in this new technological era.

In conclusion, the evolution of technology is unstoppable and is leading us to a drastic change in the way we interact with technology. In the not too distant future, we will be able to see students in class doing their practices “typing in the air” and it will seem normal to us. But, beyond this immediate revolution, an even more radical horizon is glimpsed: brain-computer interfaces. Although today augmented and virtual reality are rewriting the rules of interaction, BCIs could represent the next great leap, allowing direct communications between our brain and machines. This future, while fascinating (and dystopian), also raises deep ethical questions about privacy, autonomy, and identity in a world where our thoughts may not be just our own. For now, we are left with the future that we can foresee and manage, but the debate about what comes next will certainly provide much to talk about.