Haptic Feedback: Definition and Applications

Abstract

Haptic feedback refers to the use of touch and vibrations to communicate sensations or feelings to a user. Here in this paper we are going to dive deep into what is haptic feedback and how this technology is being implemented in VR systems.

Introduction

I. INTRODUCTION

The word haptic comes from a greek word which means "touch". Feedback means information which we receive, hence haptic feedback is the feeling of touch which we receive back.

Mobile phone vibration is very often described as an example of haptic feedback technology. But it is just one and very simple illustration of how haptics can function. Haptics allows a user to interact with computer-based devices by receiving tactile and force feedback.

Now based on usage, haptic feedback can be of three types-

1. Graspable(joysticks)
2. Touchable(used in mobile phones)
3. Wearable(VR gloves)

But based on feedback, haptics can be of five broad types-

• Force (This form of haptics originated in the late 1960s, making it one of the oldest and most well-studied types of this technology. Force feedback is the simulation of real-world physical  touch, while in a simulated environment such as virtual reality (VR), by way of motorized motion  or resistance.)

• Vibrotactile (Vibrostimulators apply pressure to the definite receptors of human skin.)
• Electrotactile (Electrotactile stimulators affect both receptors and nerve endings by applying         electrical impulses. By means of electrical impulses, a user can receive a wide range of sensations         which cannot be reproduced with any other current feedback systems. Moreover, electrical         signals are the basis of the nervous system, so we can say confidently that this type of haptic feedback is the best suitable for generating and simulating the real-world sensations.)[1]
• .Ultrasonic (it turns ultrasound into virtual touch. The place where all the ultrasound waves         coincide is called the focal point. Where the focal point is positioned in 3D space is programmable         in real time. It can change position from instant to instant. We will use a hand tracking device to         track the exact position of our hand and position the focal point at a spot on it.)

• Thermal (Thermal feedback is the ability to sense temperature and the flow of heat energy. It can also refer to haptic feedback that simulates temperature changes on the skin, such as when a  user touches something hot or cold. Thermal feedback devices use actuators to move heat around the skin)

II. USES OF HAPTIC FEEDBACK OTHER THAN VR

We will be focusing more on how Haptic technology is being implemented in VR but first let's see where haptic technology is being used.

1. Mobile and Computers
2. Automotive and Aviation
3. Healthcare
4. Gaming

We won't go into much details on how haptics are being implemented in mobile, computer devices in this paper, but the other points are specified in details as below.

III. HAPTIC FEEDBACK IN VR

In VR systems, to increase the levels of immersion, we use haptic feedback. Now let's see an example why we need haptic feedback. For example we place a mug in a table in real life. Don't we feel it?

Similarly in VR system, when I put this mug on the table I need to feel a small vibration the moment the mug touches the table. This can be supported by the controller in VR. Because we have the position tracking of the controller, the system knows where the mug is moving in a 3D space, and it knows the 3D position of the table. And when these two overlap with each other in the 3D space, the moment the mug touches the table, the system can detect it and make the controller generate a small vibration to resonate with what I expect to feel in real life.

IV. HOW HAPTIC FEEDBACK IS CREATED

Haptic Feedback is all about forces, vibrations and movement. The most common way is by using an eccentric rotating mass(ERM) actuator. As the ERM rapidly spins, the force from the weight becomes unstable, causing the motor to move. The moved motor is what creates the haptic feedback.

ERM’s were followed by linear resonant actuators (LRA’S), which consist of a magnet attached to a spring, surrounded by coil and protected by an outer layer. The electromagnetic coil is energized so that the magnetic mass moves back and forth within it, creating the sensation of the feedback. The Nintendo Switch uses a LRA actuator to give gamers a more immersive experience.[2]

Due to the demand driven in different application fields, various 3-DoF haptic rendering algorithms have been studied, including rigid body interaction, elastic body interaction, fluid simulation and topological changing force interaction. Obviously, the 3-DoF haptic rendering algorithms cannot simulate the multi-region contacts between objects. When the tool avatar contacts with objects with multi-region contacts, the avatar will penetrate the obstacle, which greatly reduces the simulation fidelity. In addition, 3-DoF haptic rendering algorithms cannot simulate the interactive torques between objects, which results in unrealistic haptic sensation.

With the advent of 6-DoF desktop force feedback devices and the realistic force interaction requirements for simulating complex scenes, Puterbaugh et al. from Boeing Inc. first proposed the concept of 6-DoF Haptic rendering in 1999. The aim is to solve the problem of multi-point and multi-region interaction simulation with non-penetrating contacts between complex shape objects, and the computation of six-dimensional force and torque for simulating rigid body interaction. Various 6-DoF haptic rendering algorithms have been studied due to the demand driven in different application fields. In 2008, Lin and Otaduy published a book that systematically summarized the work of several well-known scholars in haptic rendering field. Up to now, many problems remain to be solved for achieving realistic 6-DoF Haptic rendering. Based on the collision response algorithms, 6-DoF haptic rendering algorithms can be classified into penalty-based and constraint-based approaches. In the penalty-based approach, contact constraints are modeled as springs whose elastic energy increases when the haptic tool penetrates into an object in the virtual environment. Penalty forces are computed as the negative gradient of the elastic energy, which pushes the graphic tool toward a non-penetrating configuration. As pointed by Ortega et al, these methods allow the virtual objects to interpenetrate, which may lead to missing collisions and the well-known pop-through effect. In contrast to the penalty-based approach, constraint-based approach is an analytical and global method for the collision response. Constraint-based approaches attempt to constrain the pose of the graphic tool to be free of penetration, while the haptic tool can penetrate into objects. Constraint-based methods achieve accurate simulation by using a linear complementary problem model for the normal and friction contact constraints, which is computationally expensive and cannot handle large number of multi-region contacts between complex-shaped objects or large-scale deformation at the kHz haptic rate. To solve this problem, Wang et al. proposed an efficient constraint-based rendering algorithm using configuration-based optimization and hierarchical sphere-tree modeling.[3]

An application of desktop haptic devices is virtual surgery which we will know more about in the next section. 

Now another way of generating haptic feedback is by wearable vests. A haptic feedback suit is a suit that provides feedback in the form of a vibration. For example, when we are shot in a VR game, we actually feel a vibration where we are shot.

In 1994, Aura Systems launched the Interactor Vest, conceived by Aura's VP of Audio and Video Technologies, Larry Shultz to feel sound from video games and TV shows. The Interactor was a wearable force-feedback device that monitors an audio signal and uses Aura's patented electromagnetic actuator technology to convert bass sound waves into vibrations that can represent such actions as a punch or kick. The Interactor vest plugs into the audio output of a stereo, TV, or VCR and the user is provided with controls that allow for adjusting of the intensity of vibration and filtering out of high frequency sounds. In January 2002, Francesca Rosella and Ryan Genz, then researchers at the Interaction Design Institute Ivrea in Italy, designed the HugShirt. The Hugshirt is a wearable haptic telecommunication device that allows a wearer to send the feeling of a hug to a distant loved one. HugShirts feature touch sensors and haptic actuators that work together to capture and recreate touch over distance. Sensor areas placed on the garment capture the touch of the wearer, the data is transferred to their mobile device where the Hug App creates a Hug message that is delivered to the receiving wearer of a second HugShirt in another location across the world. The SoundShirt is a shirt that allows deaf and hearing audience members to experience music and AR enhanced by touch (haptic) sensations. The SoundShirt was used for its first performance by the Junge Symphoniker Orchestra in Hamburg, Germany. During a live or virtual performance the shirt maps different musical sounds to haptic sensations on different parts of the body, allowing media to be felt physically. The SoundShirt features 30 haptic multi-force actuators embedded into a garment. Finally came the Teslasuit which induces electrostimulation, motion capture, and biometry to provide realistic immersion.

V. HAPTIC FEEDBACK IN HEALTH CARE SYSTEMS

Haptic feedback is used in various healthcare systems using a VR system. For example, it can be used to  remotely examine patients or perform robot assisted surgery. Force Feedback Plays a Significant Role in Minimally Invasive Surgery, where an automated laparoscopic device is used with force feedback capability to help surgeons differentiate tissue stiffness easily through a haptic interface device.[4]

Haptic feedback can also be used for rehabilitation and physical therapy. For example, Prosthetic limbs can be designed to replicate the sensation of touch, while visually impaired patients can experience the world through haptic feedback since with haptic feedback, they can “view” and comprehend digital information that was previously inaccessible to them [5]

Haptic technology can be used to train medical professionals as it allows them to experience the physical nature of a procedure virtually. Some examples of how haptic technology is being used are shown below:

[i] Haptic Vibrotactile Trigger technology used in stroke rehabilitation to help patients regain their motor skills and improve the range of motion in their limbs. The technology works by providing patients with feedback on their movements through vibrational pulses, allowing them to make corrections as needed.

[ii] Surgical Planning Training System is an innovative new way for medical students to practice and refine their surgical techniques. With its realistic haptic feedback, users can feel the texture and resistance of tissue as they practice various common surgical procedures. This helps them develop a better understanding of their tools and how to use them most effectively. By accurately simulating a variety of scenarios, the ImmersiveTouch system offers a realistic, safe, and cost-effective way for medical students to become more proficient in their surgical skills. Additionally, medical professionals can use the system to reinforce their existing skills and stay up to date with new procedures.

[iii] Sound Awareness wristband provides real-time haptic notification of sound events for hearing impaired individuals. This is accomplished through a combination of microphones and sensing algorithms that detect and classify sounds, combined with haptic feedback from wristbands or other wearable devices to notify the user when a sound of significance occurs. For example, the technology could be configured to recognize and notify the user of audible warning signals such as fire alarms and car horns.

[iv] Medrobotics is an advanced robotic system that enables surgeons to feel and better control their instruments during minimally invasive surgery. With this technology, surgeons can feel the delicate tissue they are working with and reduce the risk of accidental damage. Additionally, its force feedback capabilities allow for more precise control of instruments, allowing for quicker, smoother completion of procedures with less fatigue on the part of the surgeon. [6]

Hence we can conclude that haptic feedback has huge potential in improving healthcare services.

VI. HAPTIC FEEDBACK IN AVIATION

The journey of haptic technology in aviation, though relatively recent, is marked by significant milestones that have shaped its current state. The initial exploration into haptic technology in aviation dates back to the late 20th century, when researchers and engineers began to understand the potential of tactile feedback in enhancing pilot training and flight operations. One of the earliest applications was in flight simulators, which sought to replicate the physical sensations of flying as accurately as possible. The goal was to create a training environment that mimicked the real-world flying experience, especially the tactile feedback from control surfaces and flight deck instruments. These early simulators, while primitive by today's standards, laid the groundwork for future advancements. They employed basic haptic mechanisms, such as force feedback systems, to simulate the resistance and pressure pilots would feel in an actual flight deck. For example, a pilot training on these simulators could experience the force on the yoke during a simulated high-speed turn or feel the vibration feedback indicative of different flight scenarios.

As technology advanced, so did the sophistication of haptic systems in aviation. The evolution of haptic interfaces in aviation contexts is a testament to the relentless pursuit of more immersive and realistic flight training and operational tools.

In recent years, the advancements have been particularly noteworthy. Modern haptic systems in simulators now offer highly detailed and nuanced feedback, closely replicating the sensations of various flight conditions, such as turbulence, runway textures, and even the sensation of landing gear deployment. Moreover, the evolution of haptic interfaces is not confined to training alone. In the operational realm, these systems are being integrated into actual aircraft flight decks. For instance, the development of touch-sensitive, haptic-enhanced controls in newer aircraft models is a significant leap forward. These controls provide pilots with tactile cues and vibrations, offering an additional layer of sensory input that can be crucial during complex maneuvers or in response to changing flight conditions.

This historical progression of haptic technology in aviation reflects a broader trend within the industry: the continuous striving for enhanced safety, precision, and realism. By understanding where haptic technology started and how it has evolved, we gain a clearer picture of its transformative impact on aviation training and operations. In the next sections, we will delve deeper into the technical aspects of these systems and explore their practical applications in both training and real-world aviation scenarios.

Technical Aspects of Haptic Technology and Types of Systems:

Haptic technology in aviation is a multifaceted domain, comprising various systems designed to enhance the tactile experience of pilots. Primarily, these systems can be categorized into tactile and force feedback types. Tactile haptic systems are designed to simulate the sensation of touch or texture, providing cues through vibrations or other means directly to the pilot's skin. This type of feedback is crucial in simulating scenarios like turbulence or the feel of different terrains during landing. On the other hand, force feedback systems are more concerned with simulating the resistance and forces a pilot would encounter when manipulating aircraft controls. In a real aircraft, pilots feel resistance from the control yoke when executing maneuvers, which is vital for intuitive control of the aircraft. Force feedback systems in simulators or modern fly-by-wire aircraft replicate this physical resistance, providing a crucial sensory cue that enhances the pilot's control and understanding of the aircraft's behavior.

The key components of haptic systems include actuators, sensors, and control algorithms. Actuators are devices that produce physical movements or forces, based on electrical inputs, to create tactile or force feedback. They can vary from simple vibrating motors in tactile systems to complex mechanical arrangements in force feedback systems that simulate the push and pull on aircraft controls. Sensors play a critical role in haptic systems by detecting the user's actions and the environment's response. These sensors then relay information to the control system, which uses sophisticated algorithms to determine the appropriate type and level of feedback. This feedback loop is crucial for the realism of the haptic experience, ensuring that the system's responses are as life-like as possible.

Integrating haptic technology into aviation systems requires meticulous design and engineering. In training simulators, haptic feedback systems are integrated with advanced visual and auditory simulation systems to create a fully immersive training environment. This integration allows pilots to experience a realistic flight deck environment, where every control touch, movement, and aircraft response is closely mimicked. In the operational context, integrating haptic feedback into aircraft systems involves working with the aircraft's existing control mechanisms. For example, in fly-by-wire systems, where physical controls are replaced with electronic interfaces, haptic feedback is essential to provide the necessary sensory cues to pilots. This integration is carefully calibrated to ensure that the feedback is not just realistic but also precise and reliable, as it directly impacts the safety and performance of the aircraft.

The technical sophistication of haptic systems in aviation underscores the industry's commitment to leveraging cutting-edge technology for enhanced safety and efficiency. As we explore the applications of these systems in training and operations, their importance in modern aviation becomes increasingly evident.[7]

VII. HAPTIC FEEDBACK IN GAMING

Haptic feedback in gaming uses physical stimuli like vibrations to simulate tactile experiences, making the game feel more realistic and immersive. The goal is to make the player feel more connected to the game and feel like they're not just playing it, but also in it. Haptic feedback can provide instant feedback on player actions, such as the recoil of a gun when firing, and can correspond with different visual and audio stimuli.

A modern game controller has the functionality to express haptic feedback, also called “rumble”. These controllers produce rumble by one or several builtin haptic motors which can shake the controller at different frequencies.Haptic motors were first introduced for video game controllers with the Nintendo 64 in 1997 following the release of the Rumble Pak add-on. Since then, the use of haptics in modern controllers has been following the same design philosophy, with two haptic motors located close to each palm of the player's hands.

A haptic feedback vest was first presented in 1994 but it has never been included as a standard part of video game equipment; standard haptics for games comes through the controller.[8]

Another way to feel haptic feedback is through haptic gaming chairs. The chair's advanced haptic engine is capable of simulating a full range of vibrations, textures, and motions, in turn generating tactile feedback that’s both realistic and responsive.

VIII. HAPTIC FEEDBACK FUTURE

In short, haptic technology will keep growing as there are still a lot of limitations we need to overcome.

The Future of Haptics in immersive gaming and healthcare using VR systems. As more and more immersion technologies will be developed, the more haptics will also develop.

Haptic technology is already used in many devices, including smartphones, laptops, and robots, and the global market for it is projected to grow from $16.8 billion in 2022 to $47 billion by 2030.

Haptic devices can be grouped into three main types: graspable, wearable and touchable.

Graspable includes joysticks with can be used for operation of surgical robots. Numerous studies have shown that adding haptic feedback to the control of these robots increases accuracy and reduces tissue damage and operation time. Getting a feel for what the robot under your command is doing would also be helpful for defusing bombs or extracting people from collapsed buildings. Or for repairing a satellite without suiting up for a spacewalk. But “fooling the brain only goes so far,” says Ed Colgate, a mechanical engineer at Northwestern University who works in haptics. It’s sometimes easy to break haptic illusions. To his mind, in the long run engineers will need to recreate the physics of the real world — weight and all — as faithfully as possible. “That’s a really hard problem.” Graspable devices often take advantage of kinesthetic sensations: feelings of movement, position and force mediated by nerves not just in our skin but also in our muscles, tendons and joints. Wearable devices, on the other hand, usually rely on tactile sensations — pressure, friction or temperature — mediated by nerves in the skin.More commonly, wearable haptic devices communicate through vibration. Culbertson’s lab, for example, is working on a wristband that guides the wearer by vibrating in the direction he or she needs to turn. And NeoSensory, a company founded by Stanford neuroscientist David Eagleman, is developing a vest with 32 vibratory motors that was showcased in an episode of HBO’s sci-fi series Westworld where it ostensibly helped characters identify the direction of approaching enemies. One of the vest’s first real applications will be to translate sound into tactile sensation to make spoken language more intelligible to people with profound or complete hearing loss. Eagleman is also working on translating aspects of the visual world into vibrations for people who are blind.[9]

Touchable haptic devices are found in mobiles screens which gives a sense of vibration when touched.

Touchable surfaces also allow types of illusions. For instance, Culbertson says, playing the sound of a button clicking as one taps a picture of a button makes it feel as if the button is actually clicking. Or making the screen appear to deform under one’s finger can make it feel softer. People construct perception by tying together sight, sound, touch, taste and smell — and, as Culbertson says, “It’s really easy to fool your brain if you have a mismatch between your senses.”[9]

In either case, very soon movies like Ready Player One will be the new reality as haptics and VR technology keeps developing.

References

[1] https://teslasuit.io/blog/haptic_feedback/ [2] https://www.dbox.com/en/haptictechnology#:~:text=In%20short%2C%20haptic%20technology%20creates,something%20that%20contains%20haptic%20technology. [3] https://www.sciencedirect.com/science/article/pii/S2096579619300130 [4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1356852/ [5] https://www.scirp.org/journal/paperinformation?paperid=122927#:~:text=Haptic%20technology%2C%20which%20involves%20the,for%20doctors%20to%20perform%20procedures. [6] https://medium.com/@TheImmersiveNurse/healthcarehaptics66a12dd9d7d7#:~:text=Through%20haptic%20technology%2C%20doctors%20and,see%20even%20more%20exciting%20developments. [7] https://www.linkedin.com/pulse/feel-flight-haptic-advances-aviation-elias-double-a-andrews-aj1jc/ [8] https://dl.acm.org/doi/fullHtml/10.1145/3552327.3552333#:~:text=A%20modern%20game%20controller%20has,the%20controller%20at%20different%20frequencies.

Copyright

Copyright © 2024 Rohan Dutta. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.