State Sensor Investigations and Operator State Classification and Feedback Algorithms

Obejctives:

Identify key operator states to monitor
Identify technologies for operator state assessment
Evaluate sensors, methods, and approaches for real-time assessment of operator states
Evaluate operator state integrated calssification methods

Problem:

Avionics design does not currently take into account an operator's real-time cognitive abilities and limitations. The industry could design methods that sense and react to cognitive overload situations. Development of these tools could improve products that can assist pilots by providing them with key information based on real-time needs.

Research Methods:

OPL is proud to conduct this research in collaboration with NASA's Dr. Alan Pope and Kara Latorella. The project is funded by the NASA Langley Integrated Intelligent Flight Technologies.

Discovery NRC - The NRC Helicopter demonstrates the collaboration with NRC of Canada with the study of the operational environment of a Bell 412 helicopter. OPL thanks Stephan Carrignan and Dr. Todd Macuda from NRC for the opportunity to perform cutting-edge research on their Bell 412.

Computerized Airborne Research Platform (cARP)

The Computerized Airborne Research Platform (CARP) was designed so that human factors could be studied along with advanced flight deck technology. The airplane is a Beech A-36 Bonanza, and it is equipped with sophisticated equipment that was designed at OPL. The equipment is loaded onto the plane by being placed on a rack. OPL is currently developing a tool called "Operator State Characterization and Feedback (OSCAF)" to study physiological data. This tool allows for spatial and spectral analysis of bio-electrical signals.

Physiological Sensors

OPL is using the following technologies to determine which physiological characteristics are important for real-time pilot state assessment:

Indigo Thermal Camera	The Indigo Thermal Camera allows for the facial features of the pilot to be studied. It is used in conjunction with the SmartEye system to provide information on the location of important facial features.
SmartEye IR Camera	The SmartEye system is used for eye tracking and head orientation and can assist in providing information to be used for thermal analysis.
128-Sensor high density EEG	The EEG can provide information on the physiological state of the brain by measuring brainwaves.
Electro-cardiogram (EKG)	EKG analysis can measure the electrical waves of the heart.
Respiration Belt	The respiration belt can keep track of the breathing rate of the pilot.
Electro-oculargram (EOG)	The EOG is used to measure the resting potential of the retina. This technology can be used to measure the position of the eye, as well as how it adjusts to differences in light.
Temperature sensors	The temperature of the pilot's skin can be measured using temperature sensors that are underneath the pilot's suit.
Pulse Oximeter	This device can report the amount of oxygen present in a pilot's blood. The oximeter uses LEDs to determine the ratio between oxyhemoglobin and deoxyhemoglobin.
Galvanic skin response (GSR)	This technology can assist in determining the emotions of some pilots.

Cognitive Avionics Toolset (CATS)

The Cognitive Avionics Toolset is a powerful software program that is used to provide real-time data exploration and analysis to support effective operator state feedback. The toolset has been designed so that OPL can incorporate EEG, EKG, EOG, respiration rate, GSR, Thermal Cameras, Facial Features, Pulse Oximeter, Flight State, and Eye Tracking. The program also allows users to "relive" the events that the aircraft undergoes in a virtual setting. The software even has the ability to recreate the instrument panel.

Sponsors and partners

NASA

projects:

Neural Avionics
State Sensor Investigations and Operator State Classification and Feedback Algorithms
Rotorcraft Brownout Study
Synthetic Flight Bag (SFB)