Molecular manoeuvring with gesture and voice

Researchers can now learn how to control 3D representations of molecules with their hands using a new piece of software, the Molecular Control Toolkit.

Developed by scientists at Sydney’s Garvan Institute of Medical Research and CSIRO, the toolkit connects commercially available gesture and voice recognition devices, such as LeapMotion or Kinect, with any molecular graphics system.

Visualising the structures of complex molecules is key to understanding their function, modelling their behaviour and, for example, designing better drugs.

3D gesture control is a standard feature of many computer games and may become as pervasive as using a mouse or touchpad.

By connecting gesture and voice recognition devices and graphics systems, the toolkit allows researchers to use hand movements to rotate a molecule, change the centre of rotation, zoom into and through a molecule, or select a structure of interest.

“To our knowledge, this is the first gesture toolkit for molecular graphics that supports multiple devices. It also has a set of controls sufficiently rich to be useful in the day-to-day work of a broad range of life scientists,” said team leader Dr Seán O’Donoghue, who works at both Garvan and CSIRO.

Previously, to carry out these functions, researchers first had to learn complex keyboard and mouse combinations, whereas the toolkit allows intuitive manipulation of a molecule after a relatively short instruction period.

The toolkit was tested on 18 life scientists working at the Garvan earlier this year. All scientists had high computer literacy but little experience with 3D gesture control. Each person was shown what to do and then given a few simple tasks to perform. Most participants grasped the basics within half an hour.

Professor Mike Rogers, biochemist and bone researcher, who took part in this pilot study, said, “It would be a great tool for giving lectures or presentations, allowing you to manipulate molecules easily with your hands. For example, it’s a very visually powerful way of highlighting the parts of a protein that might be important for drug binding.”

A pre-print of the paper is available in the IEEE Digital Library.