HUMAN
PERFORMANCE 2008 Abstracts
Student Flight Instructor Competencies
Training Interventions for Reducing Flight
Mishaps
A
Methodology for Simulation-based Job Performance Assessment
Combat
Veterans Use of FBCB2: Lessons for Training
Managing Knowledge of Human Performance
Assessment: A Human Sigma Approach
Developing Team Performance Models: From
Abstract to Concrete
Emerging
Concepts in Interagency Coordination Training
The
Worst that Can Happen: Creating Realistic Emergency Management Scenarios
Student Flight Instructor Competencies
2008 Paper No. 8246 Patricia
C. Fitzgerald, Dee H. Andrews Air
Force Research Laboratory Mesa,
AZ Brent
Crow Consortium
Research Fellows Program Mesa,
AZ Merrill
R. Karp, Jim Anderson Arizona
State University Mesa,
AZ The research literature addresses
a variety of questions concerning flight instructor training, however, more
research is needed to elucidate the instructional competencies associated with
successful instruction in this critical field. This paper presents
observational research to identify flight instructor competencies and
patterns of instructional behavior. Flight instructor behaviors were defined
in a computer-based observational tool that allows behaviors to be logged.
Seventeen Certified Flight Instructor Instrument (CFII) students were
videotaped as they were instructing Instrument flight students on a flight
simulator. The researchers coded the student’s behaviors using an observational
data collection tool. Observed behavioral patterns are presented. The
identification of critical instructional competencies during training and the
use of the computer-based behavior logging tool in training flight
instructors is discussed. Follow-on studies to further investigate methods of
enhancing instructor performance are presented. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
Training Interventions for Reducing Flight
Mishaps
2008 Paper No. 8042
Increasing numbers of preventable mishaps
across all military services led Secretary Rumsfeld and all Service Chiefs to
call for a reduction in such events by 75% from 2003 levels. Most were
attributed to human error. The highly task-loaded training and combat
missions flown by fighter pilots place particularly high demands on effective
management of cockpit resources for safe and successful mission
accomplishment. While every flight training program already includes some
form of resource management training, there is surprisingly little evidence regarding
the effectiveness of varying training approaches to reduce flight mishaps. This
paper describes a project to help the Air Force reduce preventable mishaps by
determining the specific root causes of fighter and unmanned aerial system
mishaps, developing behaviorally-based training objectives, identifying
promising training media alternatives, and defining specific measures of
effectiveness. Mishap reports revealed several repeating problems in the
areas of situation awareness, task management, and decision making in all
platforms studied. A Delphi Panel of fighter, attack, and Predator pilots
reviewed and in some cases, amplified the specific underlying human factors
that are most challenging to pilots in tactical environments. The panel also considered
the feasibility and probable value of nine potential training interventions.
The Predator community was chosen for implementation and assessment of four
interventions – focused academic training, interactive case histories,
game-based multi-task practice, and a laptop-based simulator for team
training. A review of historical Predator student records revealed that many
trainees have difficulty mastering attention management, task prioritization,
selecting a good course of action, and crew coordination. Spiral
implementation will enable the contributions of each intervention to be
assessed using a controlled experimental design at an operational training
unit. Anticipated benefits include increased student situation awareness,
more effective task management, and improved decision making in subsequent
flights, all contributing to the ultimate goal, fewer mishaps. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
A Methodology for Simulation-based Job
Performance Assessment
2008 Paper
No. 8204
Job performance measurement is of critical
importance to any organization’s health. It is important not only to
recognize and reward good performance, but also to groom future leaders.
Developing effective assessment techniques that are valid, effective and fair
is an ongoing challenge. Assessing factual knowledge using multiple-choice
test batteries relatively inexpensive and tends to be commonly used. Hands-on
assessment is the most effective in assessing task proficiency but is very
resource intensive and expensive. Computer-based simulations provide an
alternative where users can be assessed in the context of skill application
under controlled conditions. However, simulations are expensive to produce
and maintain. Validated guidelines and methodologies are needed to help
organizations develop effective assessment simulations. In this paper we
present a standard, prescriptive methodology for developing simulations for
job performance assessment. We then describe a performance assessment
simulation for Light-Wheeled Vehicle Maintenance constructed according to
this methodology. This simulation includes automated assessment methods that
borrow heavily from existing work in intelligent tutoring systems. Finally,
we discuss future research directions based on the results of this initial
methodology and assessment. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
Combat Veterans Use of FBCB2: Lessons for
Training
2008 Paper No. 8226
This paper provides the results of survey
data from combat veterans on the use, training needs, and mission criticality
of the Force XXI Battle Command Brigade and Below (FBCB2) digital system. The
results were used to characterize FBCB2 use and to identify those procedures
that are important for design of effective and efficient future training.
Overall, combat veterans reported using fewer than half of the essential FBCB2
procedures in each phase of operation. Soldiers also indicated that about
half of the FBCB2 procedures needed greater training emphasis and that 30% of
the procedures were critical to mission success. Most importantly, the
results suggested the joint contributions of training and experience on FBCB2
efficacy. FBCB2 procedures needed to be used in combat in order for efficacy
to be indicated, but experience alone without formal training was not
sufficient to produce the highest levels of FBCB2 efficacy. The findings also
supported the idea that as people become more familiar and comfortable with digital
systems, they feel better equipped to explore different functionality. In the
case of FBCB2, it appeared that the more a Soldier knew about FBCB2 and the
longer he employed the system, the more likely he was to learn and to use new
procedures that can benefit operations. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
Managing Knowledge of Human Performance
Assessment: A Human Sigma Approach
2008 Paper No. 8031
The
Department of Defense (DoD) has adopted Six Sigma
as the basis for a disciplined methodology to improve its processes. To reduce variability of human processes is
a challenge introduced by “Human Sigma” advocates. It focuses on adapting Six Sigma discipline
to measuring and improving human competencies. DoD is one of the
largest employers, with an ongoing need to measure and assess human
performance effectiveness in the context of mission accomplishment. Accurately measuring performance, although
labor-intensive and expensive, is most importantly, science-driven. Conserving valuable resources is at the
heart of current knowledge management efforts using human performance
metrics. With shrinking budgets and
workforces, and the complexity of new technologies and warfighting
environments, the time is ripe for instituting such an initiative. This paper illuminates efficiencies,
cost-savings, and quality to be gained through developing a knowledge
management system (KMS) for human performance assessment (KMS-HPA) for the
military. We examine recent United States and International efforts to
capture and manage knowledge about measures and their development, from such
domains as air traffic control, human factors, and command and control. We examine lessons learned as they apply to
enterprise level initiatives, since no such capability exists to support the
evaluation of warfighter performance. As complexity of future military operations
escalates, and as multinational militaries unite in common operations, we
face new and unknown challenges for measuring mission success. Improvements
in processes for measuring performance effectiveness should enhance future
readiness. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
Developing Team Performance Models: From
Abstract to Concrete
2008 Paper No. 8318
As the military has moved increasingly
towards distributed networked environments for Command and Control
Intelligence, Surveillance, and Reconnaissance (C2ISR) missions, teams often
operate remotely, and decision-making is distributed. Traditionally team
training involved human observers for performance assessment, diagnosis, and
after-action review and other training intervention. However, with much of
the communication and coordination happening electronically, key aspects of
the interactions between team members are no longer accessible to these
trainers. Analyzing these communications involves poring over high volumes of
raw electronic data. This is infeasible in all but the smallest scales of
operation. Intelligent automated performance assessment tools can be valuable
cognitive aids to trainers and assist them by warehousing and analyzing team
interaction data, and presenting it to them in a user-friendly manner for
real time coaching and after-action review. In order to build such a system,
it is important to first define a concrete model of team behavior for the
domain and to define rules to assess team performance dimensions from
observations of team behavior in training exercises. Research literature is
rich with different models of team performance; however, these models are
defined at a very abstract level and not directly useful at the level of
specificity that would be needed by a rule-based artificially intelligent
assessment tool. This has always been the challenge of artificial
intelligence. In this paper, we will present a case study that shows the
process of translating an abstract team performance model into a concrete
model and the resulting performance assessment rules that can be used by an
automated tool. The model is being developed to serve as a basis for an
automated after-action review tool to support large team training exercises
within the Marines in the area of combined arms. The paper will also discuss
the lessons learned along the way. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
Emerging Concepts in Interagency
Coordination Training
2008 Paper No. 8082
The working relationship between
the military, U.S. government agencies, and non-government aid organizations
can be a difficult one. The differing cultures of each organization have
evolved out of contrasting missions and activities, resulting in different
values, modes of interpersonal interaction, and approaches to work.
Multi-cultural collectives have known coordination problems (e.g., Burke,
Hess, Priest, Rosen, Salas, Paley, et al., 2005), but most efforts to enhance
cross-cultural coordination do not take a comprehensive approach that
develops individual and collective knowledge and behavior (Roberson, Kulik, & Pepper, 2003). Our team researched the
requirements for developing interagency coordination at the field level
during stability, security, transition, and reconstruction (SSTR) operations.
This paper describes our findings and their implications for designing a
computer-based interagency planning environment. We found that
conceptualizing interagency collectives as multi-team systems (Mathieu,
Marks, & Zaccaro, 2001) provided a theory-based
method for identifying what must be developed in order to achieve successful
interagency coordination. The multi-team transition (or planning) phase may
be characterized as an interests-based, multi-party negotiation – a
collaborative problem solving task in which innovative solutions are sought
through consensus building. The success of multi-team planning or interagency
consensus building, in turn, is mediated by general strategies for success,
including interpersonal relationship building and cross-cultural
communication. We determined that the capabilities of web-based knowledge
management systems and latent semantic analysis, an automated text analysis
technique, can be integrated into a comprehensive training system that
addresses individual and collective knowledge and behavior. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |
||||||||
The Worst that Can Happen: Creating
Realistic Emergency Management Scenarios
2008 Paper No. 8189
In the typical scenario development process,
learning objectives center on performing tasks according to established
procedure. However, emergency managers must make effective, life-saving
decisions in fastpaced, rapidly changing,
ambiguous, and uncertain situations while acting within legal, cultural, and
social constraints. High-stakes critical incidents are infrequent, thus
decision makers seldom have opportunity to gain real-life experience.
Instead, they gather experience through simulated exercises that immerse the learner
in authentic, realistic situations. Generating realistic, cognitively
relevant scenarios that meet the often conflicting objectives found in
emergency events requires a combination of psychological research methods and
instructional design practices. In this paper, we present our methodology for
creating management level emergency scenarios for a computer based,
multi-player simulation-training program using a combination of standard
instructional design practices, Cognitive Task Analysis (CTA) and computer-based
scenario development techniques. We discuss the use of CTA to capture the macrocognitive functions and processes decision makers
use in actual events, the analysis of this data to understand critical
decision points, actions, and strategies, and the transfer of this
information into a computer-based multi-agency training simulation. We will
discuss a project in which we designed scenarios to exercise emergency
managers who respond to airport emergencies. The goal of this work, sponsored
by the Airport Cooperative Research Panel (ACRP), was to streamline
decisions, improve communication between and within agencies, and increase
effective response during critical airport incidents. This
paper is available on the 2008 I/ITSEC CD ROM. Order
the paper from I/ITSEC’s Website. |