Human
performance
Measuring Pilot Knowledge in Training:
The Pathfinder Network Scaling Technique
Assessing Knowledge Acquisition in
Academy Flight Screening (AFS)
Exercise Pacific Link 2:
Distributed Training for Air Battle Managers
Birds of Prey: Training Solutions to
Human Factors Issues
PACERS: Platoon Aid for Collective
Employment of Robotic Systems
Profiling is Politically ‘Correct’:
Agent-Based Modeling of Ethno-Political Conflict
Modeling Human Perception of
Situation Awareness During Constructive Experimentation
Automated Scenario-Based Training
Management: Exploring the Possibilities
A Quantitative Measurement of
Presence in Flight Simulators
The Appropriate Loudness for Flight Simulator Voice Communications
2007 Paper No.
7293
L-3
Communications
Mesa,
AZ
Arizona State
University Mesa, AZ
Researchers have attempted to
measure pilot knowledge and changes in knowledge, in both simulated and live-fly
events. However, measurement in these training environments has been more
successful in measuring overall flight performance outcomes rather than on
underlying changes in knowledge. Research to assess changes in pilots’ knowledge
as a result of training is underway at the Air Force Research Laboratory (AFRL)
in Mesa, Arizona, using the Pathfinder Network Scaling technique. The Pathfinder
method uses individual judgments of the relationships between
concepts/constructs in a domain as a basis to develop an empirically derived
representation of knowledge about the concepts/constructs. These representations
can be compared and changes in representation can be quantified to assess the
impact of an intervention on knowledge.
Previous research has demonstrated the value of Pathfinder for assessing
the impact of both education and training interventions in domains such as
computer programming. At AFRL, pilots, as part of a week-long 4-ship F-16
Distributed Mission Operations (DMO) training research program, participated in
a Pathfinder study to asses F-16 pilot understanding of complex combat mission
constructs/concepts critical to mission performance. The objective was to assess
training effects that are more fundamental and process-orientated. This paper
will report findings from a sample of 71 F-16 pilots who vary in experience
level. Our results will be discussed both in terms of practical utility of the
Pathfinder technique as a measurement methodology and in terms of knowledge
measurement as a criterion for evaluating training.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7009
Lt Abby
Barger, Lt Kamille Kemp, Capt Chad Tossell
Department of
Behavioral Sciences & Leadership
USAF Academy,
CO
The Air Force relies on the
Academy Flight Screening (AFS) program at the United States Air Force Academy
(USAFA) to screen, motivate, and prepare pilot candidates for entry into
Undergraduate Pilot Training (UPT).
In this training, cadets are taught principles and techniques used in
basic flying operations. In order
to assess the effects of this training, we compare knowledge structures obtained
from individuals that have completed AFS with knowledge structures obtained from
individuals that have not graduated from AFS. Each subject was presented 435 pairs of
basic aviation concepts and asked to rate their relatedness. These ratings were used to develop
pathfinder networks and measures of internal consistency. Results show differences in cadets’
understanding of important flying concepts. In addition, networks obtained from AFS
graduates are more similar to one another than the non-graduates. Analysis of knowledge structures using
pathfinder was also shown to be a useful adjunct to performance-based methods of
assessing training already employed in AFS.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7104
Craig Eidman, Dr Peter Crane, 1Lt Clinton Kam, Dr Sara Elizabeth Gehr, Mitch Zamba
Air Force
Research Laboratory
Mesa,
AZ
Dr Michael
Skinner, Christopher Best, Sam Hasenbosch, Eleanore Burchat, Melanie Finch, Christopher Shanahan
Air Operations
Division, Defence Science and Technology Organisation,
Melbourne,
Victoria
Distributed simulation provides
warfighters with training to enhance their team and
inter-team skills with greater frequency and at lower cost than range training
exercises. Distributed simulation training for small groups of warfighters such as a formation of four fighters working
with an Air Battle Manager can be focused on specific skills such as
beyond-visual range, dissimilar air combat tactics using constructive
simulations as adversary forces.
Training for command and control teams, however, requires interactions
among blue force entities, particularly voice communication, that cannot be
supported using only constructive simulations. One solution is to conduct large scale
virtual training events such as VIRTUAL FLAG exercises. Another solution is to combine
human-in-the-loop virtual simulators with white-force role-players who provide
responsive verbal communications for constructive entities. Both of these approaches require
participation from a significant number of warfighters
or subject matter experts which increases training cost and decreases ability to
focus training on command and control teams such as Air Battle Managers. To overcome these difficulties,
Australian and US researchers conducted Exercise
Pacific Link 2 in which an Air Battle Manager team in Melbourne, Australia was
networked with a four-ship of F-16 simulators and a constructive forces
simulator in Mesa, Arizona. Using a
novel approach to scenario design combined with an improved constructive entity
generator, a small team of pilots and engineers provided five, fully interactive
four-aircraft formations of F-16s which engaged multiple waves of adversary
aircraft over a one-hour vulnerability period. Evaluations from the Air Battle
Management team…
This paper is available on the
2007 I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7133
|
Air
Force Research Laboratory Mesa,
AZ |
Crew
Training International Memphis,
TN |
CAE-USA Shalimar,
FL |
The use of unmanned air vehicles
(UAVs) in military operations is expanding rapidly, and this trend will likely
continue given increases in funding for UAV development from $3 billion in the
1990s to over $12 billion for 2004-2009. High UAV mishap numbers have generated
multiple reviews of unmanned operations in the past few years, but even within
common platforms, different analysts attributed these mishaps to differing
causes. Thirty Air Force Predator
Class A mishaps (more than $1 million damage) occurred from the introduction of
this system into the Air Force inventory in 1995 through the end of FY 2006.
Reports were reviewed to identify trends.
Substantial changes over time were observed regarding annual mishap
rates, annual mishap counts, and causal factors. Mishap rates across the past
three years dropped to less than one half the rate
across earlier years. Mishap counts, however, steadily increased, as did
Predator flying hours. Early mishap reports typically cited mechanical problems
and operator station design issues. Mechanical problems were much less
frequently cited in the last three years.
Rather, 80% of recent mishaps cited causal human error factors. Equipment
interface problems were still cited as causal or major contributing factors in
almost half of recent mishaps. Recent mishap reports often cited shortfalls in
skill and knowledge (checklist error, task misprioritization, lack of training for task attempted, and
inadequate system knowledge), situation awareness (channelized attention), and
crew coordination. These trends come in a period characterized by a rapidly
growing crew force and highlight the need to revisit both individual and team
Predator training objectives and consider alternative training interventions
that focus on the practice and improvement of these key operator skill areas.
Predator team coordination and situation awareness training objectives are also
addressed for the command and control personnel with whom Predator crews
interact.
This paper is available on the
2007 I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7110
U.S. Army
Research Institute
Orlando,
Florida
In the future, Army platoons will
be equipped with small aerial and ground robotic systems, which will provide the
unit with real time information about the immediate surroundings; what’s over
the next hill, around the next corner, or on the roof of a building. The Army
currently is supplying platoons with both prototype and commercially available
unmanned systems to evaluate military utility. The training provided prior to
these “experiments” focuses almost entirely on individual operator training;
but, training on how to integrate the system into unit operations is also
required if the system is going to be properly exploited. While operators need
training on the details of each system, the rest of the unit needs training on
more general aspects involving coordination and communication. In addition, the
employment of robotic systems puts new responsibilities on the platoon leader in
terms of planning, delegation, resource allocation, coordination and workload,
which he or she must learn to handle. Many of these unit level training
objectives are common across different systems, both aerial and ground. The
purpose of this paper is to (1) lay out these system-general aspects and (2)
suggest a list of activities that trainers and leaders could focus on in order
to help train unmanned system employment at the unit level. For each activity,
associated observations and after-action review questions are suggested.
Trainers could use these to help assess proficiency and coach the unit in system
employment. One benefit of this system-general approach is that trainers don’t
need to stay conversant with the details of every system a unit may bring to a
training event. The benefits that this kind of training could provide will be
illustrated with examples from the Micro Aerial Vehicle Advanced Concept
Technology Demonstration 2006 Soldier Experiment.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7468
|
Department of National
Defence Ottawa,
Ontario, Canada |
CAE
Professional Services Ottawa,
Ontario, Canada |
Armoured Vehicles (AVs) have become
sophisticated pieces of modern machinery. Today’s soldier is faced with expanded
roles in AVs that greatly increase workload, tasks, and training requirements to
operate these modern vehicles.
Soldiers must now manipulate several complex systems such as fire
control, command, navigation, defensive aids and automotive systems
simultaneously as well as manage the distribution of crew tasks and maintain
overall situational awareness.
Exacerbating the problem in current AVs is that each vehicle sub-system
has separate interfaces that were developed independently by sub-system
contractors. It is clear that in
order to support the continued development of future AVs and harness the full
capabilities of these complex systems, we must consider the soldier as an
integral part of the vehicle system, and design system interfaces that reflect
their requirements. This
soldier-centered interface development process will be traced through the
outcomes and experiences of the currently running Multi-Role Combat Vehicles
Technology Demonstration Project (MRCV TDP) within Defence Research and Development Canada. This project focuses on future
requirements for the Canadian Light Armoured Vehicle
fleet and is directly linked to major capital acquisition programs funded by the
Canadian Forces. The first phases
of MRCV TDP encompassed a series of international distributed simulation
experiments using high-fidelity ground AV simulators, with highly configurable
operator-machine interfaces. These
AV simulators were linked to virtual and constructive ground and air asset
simulations, from both Canadian and U.S. research facilities. This paper examines soldier-centered
interface design, and recommends an interface development process for use in
future military vehicle procurement programs – from a Project Manager’s
point-of-view. This process of
soldier-centered interface design is crucial for the future implementation of
AVs, allowing our soldiers to effectively control these complex systems, strike
with precision, and have the …
This paper is available on the
2007 I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7015
Barry G.
Silverman, Gnana
K. Bharathy, Benjamin Nye
Electrical and
Systems Engineering Department University of Pennsylvania,
Philadelphia,
PA
A holy grail for military,
diplomatic, and intelligence analysis is a valid set of software agent models
that act as the desired ethno-political factions so that one can test the
effects that may arise from alternative courses of action in different lands.
This article enumerates the challenges of such a testbed and describes best-of-breed leader and follower
profiling models implemented to improve the realism and validity of the agent.
Realistic, ‘descriptive’ agents are contrasted to rational actor theory in terms
of the different equilibria one would expect to emerge
in conflict games. These predictions are examined in two real world cases (Iraq
and SE Asia) where the agent models are subjected to validity tests and a policy
experiment is then run. We conclude by arguing that substantial effort on game
realism, best-of-breed social science models, and agent validation efforts is
essential if analytic experiments are to effectively explore conflicts and
alternative ways to influence outcomes. Such efforts are likely to improve
behavioral game theory as well.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7448
|
M Toyon Research Corporation Goleta,
CA |
Informaton Sciences
Institute Marina
del Rey, CA |
Michael Anhalt, Jacqueline M. Curiel
Alion Science and
Technology
Marina del Rey, El Cajon, CA
Highly advanced sensor
technologies give our military commanders a significant command and control (C2)
advantage over our enemies during conflicts, particularly with respect to
situation awareness (SA). The use of advanced sensor technology models in
synthetic battlespace gives war fighters parallel advantages. Two accepted
simulation methodologies for analyzing the impact of sensor technologies are
through Human-in-the-Loop (HITL) experiments, such as Joint Urban Operations
(JUO), which utilize sensor capabilities to assist human participants during the
experiments, and Monte Carlo Constructive (MCC) simulations, which can be used
to model human performance. In HITL experiments using Joint Semi-Automated
Forces (JSAF), participants describe their SA using Situation Awareness Objects
(SAOs) which then can be reconstructed using Endsley’s
(1995) three levels of SA (perception, comprehension, and prediction). MCC
experiments, which are dominated by algorithmically determined behaviors, can be
used to model SA. Sensor measurements currently can be fused to perceive individual entities, but do not have the
capability to recognize groupings of entities, resulting only in partial
perceptual SA. Furthermore, current sensor data fusion models do not produce the
second and third levels of SA, comprehension and
prediction.
This paper will report research
efforts to utilize both methodologies to expand the use of SAOs beyond player
declarations to the automatic generation of SAOs. We develop a method to organize events
drawn from…
This paper is available on the
2007 I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7233
Bruce
Perrin, Barbara Buck, Brandt Dargue, Elizabeth Biddle, Troy Stull, Curtis Armstrong
The Boeing
Company
St. Louis, MO
USA
Despite the prevalence of
software applications that exploit user information to individualize the
experience, personalized training systems are still relatively rare. This paper describes changes in
technology and standards that may alter this trend. Utilizing these advances, we have
developed a standards-based learner model that is updated dynamically during
training and that controls content sequencing. We have established the impact of this
technology on learning through training effectiveness research. With this core learner modeling
capability established, we have subsequently started exploratory studies into
ways it might be used to manage scenario-based, simulation training. Specifically, we describe two prototype
systems that use this core modeling capability, but that use the information it
provides in distinctly different ways.
Because of the complexity of simulation training, the root cause of
performance issues is seldom apparent.
The first prototype addresses this issue by using the learner model to
select follow-on scenarios that help to build skill while distinguishing among
competing learning needs hypotheses.
The second prototype addresses the issue of maximizing learning
opportunities within a scenario. It
uses the core learner model to modify a scenario during execution in order to
provide additional opportunities to achieve specific learning objectives or to
adjust the challenge of an exercise.
Directions for future research for both efforts are described.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7031
|
Northrop
Grumman Corporation (NGC) CREOL Orlando,
Florida |
Institute for Simulation
& Training (IST) Orlando,
Florida |
College of
Optics and Photonics, University of Central Florida (UCF)
Orlando,
Florida
Virtual Reality (VR) and
Augmented Reality (AR) Head Worn Display (HWD) technology is being considered as
a novel alternative for low cost, wide Field of Regard (FOR), deployable
simulators. There are inherent
differences in display characteristics among different HWDs and between HWDs in
general with conventional displays.
For example, the effective Field of View (FOV) in most HWDs is no more
than 60º horizontal and 45º vertical, which is far narrower than the human eye’s
200º horizontal and 135º vertical FOV.
Developing a HWD with a wide 200º horizontal FOV is expensive. Current HWD flight simulator
implementations provide limited effective FOV that reduces the pilot’s visual
stimulus, perception, sense of presence and overall training effectiveness. To successfully utilize a VR or AR HWD
in a simulator, we hypothesize the user must have the same or even a higher
mental immersion experience as compared with the conventional simulator
experience with unrestricted FOV. Attempts to measure mental immersion or
presence from VR simulations were normally conducted with questionnaires. Although more convenient, the validity
of measuring the continuous experience of presence with post experience
questionnaires has been challenged (e.g., Slater, 2004). Here, a quantitative
approach to measure presence in relation to mental image processing and
performance is proposed. This paper
presents experimental methods involving measurement and analysis of normal head
and eye movement patterns of experienced pilots while accomplishing specific
tasks in a conventional flight simulator with a 170° horizontal x 75° vertical
FOR. The paper outlines metrics
taken using head and eye tracking equipment, and results of
pilot…
This paper is available on the
2007 I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.
2007 Paper No.
7150
Nathaniel M.
Napoletano, Thomas Babyak
Lockheed
Martin, STS
Akron,
Ohio
Voice communications and warning
tones are vital elements of virtually every flight-training scenario. Before and
during missions, instructors and pilots perform coordination intercommunications
and simulated radio communications across digital networks between flight
stations and other training assets. These networks may connect facilities around
the globe during U.S. Air Force (USAF) distributed mission training. In the requirements analysis phase
leading to the development of each training device, questions arise about how
loud the voice communications and warning tones should be played in the
instructor’s and pilot’s headsets. The problem is more involved than matching
the levels used in the equipment being simulated because of several complicating
factors. These factors include the facts that the pilots in the field often wear
hearing protection (earplugs) under their helmets, the ambient noise is lower in
the simulator and the training participants may be using a variety of
equipment. This study also takes
hearing safety concerns related to noise induced sensorineural hearing loss into account. The lower
criticality of voice intelligibility in a simulated mission compared to combat
alters the balance of factors considered while driving audio safety
specifications toward the more conservative. This paper defines a common set of audio
alignment parameters and methods for use by the networked fighter trainer
community. There is little existing published data establishing optimal loudness
levels particular to radio network communications taxonomies used in flight
simulators. In order to develop these parameters the authors combined publicly
available data with their new study: “Acoustical Comfort Levels in an Immersive
Training Environment.” These methods and parameters are now being used to align
pilot and instructor voice communications equipment at our facility and customer
sites.
This paper is available on the 2007
I/ITSEC CD ROM.
Order the paper from I/ITSEC’s Website.