Working with virtual hands

Digital hands offer real-life experiences 

The success of virtual training hinges on the functionality of a person’s most important tool: their hands. The more lifelike they are, the greater the learning effect. With the help of a software application, the Swedish startup Gleechi, Siemens, and Atos have now proven that virtual hands can realistically perform a wide range of tasks in industrial settings. And they can do it on a platform that allows not just individuals but entire teams to practice work processes.

Production robots can perform their tasks around the clock without complaint, but even they occasionally need a bit of loving care: for example, when the arm motor unexpectedly fails. What now? A mechatronics engineer who wants to learn more can find help in a training session using virtual reality (VR) goggles and a controller in each hand.

She sees her virtual hands in the VR goggles, and when she moves her real fingers the virtual ones perform the same movements. Easy-to-read instructions help her remove the arm, place it on a worktable, take off the metal plates, and unscrew the motor. The repairs are complicated. She has to change tools, grip the arm in different places, and lift and rotate the heavy part, sometimes using both hands. Her digital hands move so realistically that she almost forgets that she’s working in virtual space. When she arrives at the factory the following day, she can quickly repair the robot arm with the steps she practiced virtually.

Picture yourself in our Digital World

The world around us is constantly changing. At Siemens we are always challenging ourselves to build a better future. We need the most innovative and diverse Digital Minds to develop tomorrow‘s reality. Our digital careers offer the opportunity to shape the future and create impact for society. 

The next step for VR platforms

Hands are our main way of physically interacting with the world. Therefore, in many VR applications it’s helpful to use our hands, and not just when playing computer games – where today’s VR innovations are most commonly implemented – but also in virtual training. The applications range from healthcare and services to logistics and industry. These innovations are in greater demand than ever before, not least because the pandemic has forced many people to work from home. The growing shortage of experienced experts and instructors also plays a significant role, as does the limited availability of training devices, the occasional difficulty in accessing production facilities, and simply the effort involved in bringing people together in one location.

There’s an enormous need for virtual hands. Industry in particular has countless tasks that people can’t perform without using their hands, even in the foreseeable future and despite extensive automation. These tasks include wiring a commercial aircraft, checking the axle-box bearing of a railcar, setting up electronic switch boxes, and even repairing a robot arm. VR training is an option, which is why these programs are constantly being developed. “The realistic experience of virtual space, and training for teamwork in particular, are the next important advances in VR platforms,” says Georg Schöler, head of Digital Operations at Siemens. “Hands that feel just like they do in real life are a key component.”

Clear input, organic movement

Hands are a mechanical marvel: They’re delicate, highly mobile, and powerful. They allow us to literally grasp and comprehend the world. Translating them realistically into virtual space is therefore no small feat. The input devices that detect the movements of the real hands – data gloves, controllers, or scanners built into the VR goggles – need to play along. At the same time, the actions of the VR hands (like gripping and holding) need to look very realistic, because this intensifies the user’s the immersion in the virtual world and the resulting learning effect.

The Swedish startup Gleechi has developed this very type of solution. The software application VirtualGrasp generates realistic hand animations based on robot gripping technology, the particular object’s shape, and hand kinematics. In 2020 Gleechi tested and developed this solution for industrial VR training along with Siemens (as the expert in industrial applications) and the IT service provider Atos (for data analysis) as part of an EU-funded project.

Detected properties

VirtualGrasp currently works with two controllers, which allows the user to control two realistic hands. At the press of a button, they can grip an object and drop it when the button is released. The hands can also push objects and, if desired, even dribble a basketball. “It takes a bit of practice at first,” says Kai Hübner, Chief Technology Officer at Gleechi. “But experience shows that training participants quickly forget that they’re holding controllers in their hands.”

The virtual hand not only follows a command – to lift an object, for example – it also detects the object’s shape and how it needs to be gripped. When someone picks up a screwdriver in virtual space, the tool doesn’t simply stick to the hand; the fingers surround the handle just like they do in real life. These detected properties can also include weight and therefore whether the object should be lifted with one or two hands. When an area shouldn’t be grasped – for example, the metal rod of a screwdriver or an energized component – the user is immediately alerted by a red warning light.

For individuals and entire teams

VirtualGrasp can be easily integrated with just a few modifications into V@RENA, the virtual reality platform from Siemens Digital Industries, which allows companies to create VR training courses flexibly and independently: for example, to repair robot arms. However, the goal of the EU-funded development project wasn’t to support a lone pair of hard-working hands but to create VR options for training entire teams in industrial settings.

VirtualGrasp can also be implemented on a VR platform, based on the example of a two-member team who had to wire a switch box. This makes it possible to not only practice multi-part work processes but also to analyze and potentially optimize them.

A new era

Whether someone is standing alone in front of a virtual workbench or is joined by colleagues in the form of avatars on the factory floor, their hands are their most important tools, just like in the real world. VirtualGrasp will therefore be expanded and utilized. It helps that the VirtualGrasp kit can already be integrated into a wide range of VR platforms. In the near future, it will likely be used in other input devices like haptic gloves and optical motion detectors. “One thing is clear,” says Joseph Newman, AR/VR Expert at Siemens Technology. “Solutions like this mark the beginning of new era: for industrial training as well as all virtual activities.”

Hubertus Breuer, April 2021