HUMAN SYSTEMS INTEGRATION
Automated Performance Assessment
for DD(X) Post-Watch Debrief
Evaluating Training Effectiveness
During Training System Development
The Cognitive Cockpit – State of
the Art Human-System Integration
Team Exposure Stress Training
(TEST): An Approach for Reducing Stress
Team Training with Simulated
Teammates
Facilitating Leadership in a
Global Community: A Training Tool for
Multicultural Team Leaders
A Framework for Applying HSI Tools
in Systems Acquisition
Simulation and Network-Centric
Emergency Response
Current and Future Net-Centric C3:
Usage and Preferences
Measuring Situation Awareness for
Dismounted Infantry Squads
Human Factors in Air Force Flight Mishaps: Implications for Change
Automated Performance Assessment
for DD(X) Post-Watch Debrief
Ruth A. Wienclaw, Ph.D. Wienclaw Associates Paul E. Van Hemel, Ph.D. Raytheon Company In potentially life-and-death situations as may occur aboard the DD(X), it is essential to develop sound performance assessment measures, to devise and implement appropriate and thorough feedback mechanisms, and to employ feedback earlier rather than later in the process. The technology of the 21st century offers many methods to do these tasks. However, just because technology exists does not mean that its application will be useful or even that it will not have a negative impact on performance. Further, the incorporation of many technologies currently cannot be justified by their return on investment. Automated data collection and performance assessment capabilities for the DD(X) are categorized into three levels. Level 1 capabilities include automated data collection for after action review. These data will be analyzed in light of the sailor’s performance history and flagged at the watch supervisor’s station. This level may also include the ability to bookmark data and limited ability for ad hoc inquiries. Level 2 capabilities include more detailed tracking to determine why performance degradation occurs. This level may also include more advanced input technology such as voice recognition, eye tracking, and basic cognitive modeling as well as more information on the components of teamwork. Level 3 capabilities are those not feasible from either a cost or technological standpoint at the present time, including thorough cognitive modeling, ability to have immediate answers to ad hoc inquires, or inclusion of data on all components of teamwork. Whereas it is both reasonable and feasible to include such capabilities in the DD(X), they will have to be included in stages, incorporated only as technologies mature and human behavioral performance models improve. Each inclusion will need to be based on rigorous cost-benefit tradeoff analysis to ensure that the added feature’s benefits will yield an acceptable return on investment. 2005 Paper No. 2456 |
|
|
Evaluating Training
Effectiveness During Training System Development
Dennis J. Folds, Ph.D. Georgia Institute of Technology Conclusive evidence for the effectiveness of training
cannot be obtained until training actually takes place, and performance of
trainees after training can be compared to their pre-training baseline. Performance-based specifications for
training systems, however, are often worded in such terms as “shall provide
effective training”, and thereby put a burden of proof on the developer to
demonstrate that these requirements are satisfied before the system is
accepted for use. A common test method
used in these situations is to allow a few presumed subject matter experts to
view a demonstration of the training system and then provide ratings and
comments on whether the system “provides effective training.” Positive ratings are offered as proof that
the requirement is met. The validity
of this approach is obviously suspect.
A more robust test method is needed during training system
development. An analytical approach to
evaluating training system effectiveness can be applied during design and
development. Although such an approach
does not ensure that the training system will be used effectively, it can
establish that the training system has the capacity to be used effectively. This approach corresponds to other accepted
test methods for evaluating a design against requirements, when the actual
performance of the system cannot be ascertained until after development. Primary features of this analytical
approach are to (a) identify the domain of tasks intended to be trained in
the trainer, (b) identify the range of environmental and situational
conditions in which those tasks can be performed in the trainer, (c)
characterize the behavioral fidelity of task performance in the trainer, and
(d) characterize the performance measurement and feedback mechanisms for
these tasks. This analytical method
was applied to two training systems:
an embedded electronic combat trainer, and a stand-alone trainer for
tactical arms controllers. Results of
these analyses are described. 2005 Paper No. 2138 |
|
|
The Cognitive Cockpit – State of
the Art Human-System Integration
Gary S. Kollmorgen BMH Associates, Inc. CDR Dylan Schmorrow, Dr. Amy Kruse DARPA LCDR James Patrey NAVAIR The "One Team" theme concept will have to include a closer integration of system performance with human performance and limitations. The Cognitive Cockpit (CogPit) is an effort to look at leading edge physiological and neuro-physiological advances applied to operational environments, integrating the "best of the best", and providing a weapons platform capable of enhancing the human's capability beyond today's standard performance expectations. Leveraging technology advancements being matured under DARPA IPTO's Improving Warfighter Information Intake Under Stress (IWIIUS), the CogPit strives to be a revolutionary system that will provide relevant and timely operational data to the pilot in a manner that will allow increased cognitive processing while maintaining or increasing operational performance. The CogPit allows the warfighting system, the plane, to be aware of the pilot's cognitive loading and how best to provide additional or new information in a manner that will not disrupt overall system performance. Four industry teams have been maturing the IWIIUS technology over the past two years. Each team has been developing in a unique warfighting system environment and together they have encompassed the land, air and sea domains. The CogPit is looking at the different neurological and physiological sensor systems and related technology as well as cognitive performance enhancing technologies to integrate best of breed to further mature the state of the art. Additionally, the CogPit will be a test bed for developing tactical cockpits of the future. 2005 Paper No. 2258 |
|
|
Team Exposure Stress Training
(TEST): An Approach for Reducing Stress
Shatha N. Samman Global Assessment Laura Milham Design Interactive, Inc. Many military teams must operate in exceptionally stressful environments (e.g., battlefield) that include both physical (e.g., noise) and psychological stressors (e.g., time pressure). Often, exposure to such environments results in negative responses (e.g., subjective anxiety). To remedy these reactions, it has been suggested that repeated exposure to stress can decrease negative responses (Driskell & Johnston, 1998), allowing trainees to develop and perfect coping strategies. However, to date there has been little in the way of stress training intervention for teams. At the individual level, Stress Exposure Training (SET) has been used successfully to enhance affective responses to stress. It attempts to enhance the individual’s performance by providing them with an understanding of anticipated stressors and responses (physiological and behavioral) in addition to practice under stressful conditions. With practice, trainees are able to apply their coping skills while being gradually exposed to stressors. Thus, by teaching coping strategies and allowing practice sessions, trainees are able to inhibit automatic negative responses to stress and achieve higher performance levels. This same concept may be leveraged to enhance team performance under stress. This paper proposes an extension of SET to teams (Team Exposure Stress Training (TEST)), which would be aimed at enhancing a team’s ability to cope with stressors. Specifically, teams must maintain team process skills to maximize performance (Cannon-Bowers & Salas, 1998). Given this, the crux of TEST is to provide teams with knowledge on stress, define the skills and strategies needed to overcome negative effects, and practice within a stressful environment. This paper will discuss how this method can be applied to Military Operations on Urbanized Terrain (MOUT) teams. 2005 Paper No. 2376 |
|
|
Team Training with Simulated
Teammates
Emilio Remolina, Jian Li Stottler Henke Associates, Inc. Alan E. Johnston Joint team simulations are usually used to allow a team to practice working together. For example, team training simulations at the Payload Operations Center (POC) at NASA Marshall Space Flight Center (MSFC) are held in order to train a POC console position. Typically, these training simulations require instructors to help students operate the control displays, monitor and evaluate trainee’s performance and provide help and instructional feedback to students. Qualified POC operators play the role of other teammates: these teammates are given a script outlining the different interactions they will have with the trainee and other teammates and a time line for the actions they should take. Joint team simulations are however scarce and expensive. In this paper we present CITTP (Computerized Individual Trainer for Team Performance), an intelligent tutoring system (ITS) framework for individuals to rehearse their team related tasks using computer based simulations. CITTP is used as a cost effective training tool to complement team integration exercises. CITTP concentrates in defining three key elements that a team ITS must have: (i) authoring tools to define training scenarios; (ii) intelligent simulated teammates; and (iii) spoken natural language capabilities that allow simulated teammates to interact with the trainee. We illustrate CITTP when used to train a Payload Rack Officer at MSFC’s POC. A training scenario requires the trainee to apply some NASA procedures involving coordinated action with teammates. CITTP provides real time feedback, evaluates the trainee performance, executes actions on behalf of the student, and in most cases coaches the trainee whenever he makes mistakes. CITTP’s simulated teammates respond to trainee interactions by providing required information or asking the trainee for missing information. Simulated teammates also interact among themselves and change the state of the simulation by executing appropriate actions according to NASA procedures. 2005 Paper No. 2105 |
|
|
Facilitating Leadership in a
Global Community: A Training Tool for
Multicultural Team Leaders
C. Shawn Burke, Heather A. Priest, Michael Rosen, Eduardo Salas Kathleen P. Hess and Michael Paley Aptima Inc. Sharon Riedel Army Research Institute Ft
Recent events in 2005 Paper No. 2294 |
|
|
A Framework for Applying HSI
Tools in Systems Acquisition
John Burns, Jerry Gordon, Matt Wilson, Milt Stretton, & Dan Bowdler Sonalysts, Inc. Both DoD and service specific
mandates call for Human Systems Integration (HSI) to be addressed throughout
all phases of the System Acquisition (SA) process. While HSI domains and their associated
methods and tools are well defined, there does not exist corresponding
strategic or practical guidance for Program Managers (PMs)
as to when in the acquisition process different types of HSI methods and
tools have application. Absent such a
road map, PMs often rely exclusively on experience
without a basis for prioritizing cost, benefit, and data availability. A framework is presented for organizing the
strategic application of different classes of HSI methods and tools,
specifically those related to Manpower, Personnel, Training, and Human
Factors Engineering. This framework is
predicated on the idea that there are a finite set of SA strategies or use
cases for material solutions—it is salient aspects of the SA strategy that
should drive the tactical application of HSI methods and tools. It is argued that rather than defining one
process that dictates when different types of HSI methods and tools should be
applied, it is more useful to view the SA strategy as informing the
application of methods and tools in terms of a constraint satisfaction. An approach to understanding and
prioritizing constraints is presented and then using this knowledge, a
framework for application of different types of HSI methods and tools is
outlined. Example SA strategies are
detailed and the framework is applied to provide examples of when in the SA
process different types of methods and tools should be applied, what their
input requirements would be, what their output would be, and which SA
artifacts these categories of methods and tools inform. 2005 Paper No. 2281 |
|
|
Graphic User Interface Embedded
Timelines (GETs) as Visualization Tools for Distributed Instructor Teams in
Simulation Exercises
LT Gisela S. Bahr, Melissa M. Walwanis Nelson NAVAIR Beth F. Wheeler Atkinson, Eric R. Stewart JHT Inc. A principal component of the vision of future simulation based training is the collaboration of distributed instructor teams (Walwanis Nelson, Owens, Smith, & Bergondy-Wilhelm, 2003). The members of such teams are expected to coordinate and integrate the planning, control, and debriefing of an exercise while residing in physically different locations (Fowlkes, et al., 2004). In the absence of a shared team environment, we suggest that members of distributed teams will be highly dependent on a sophisticated interfacing system such as team-shared, multi-functional timelines. Thus, there is a need for specific guidance for the design of Graphic User Interface (GUI) solutions (Nosek, 2001). This paper presents solutions for GUI design based on an
analysis of the impact of distributed team environments on individual
cognition. We suggest that minimal social interaction and perceptual
limitations are likely to exert harmful effects on memory performance. Consequently, key instructor tasks
(planning, monitoring, and debriefing) are likely to be affected at the
individual instructor level and lead to performance degradation of the
instructor team. Based on a review and reappraisal of cognitive science, we
present a three-part examination, including a classification scheme for
timelines as a team-ready, low-workload GUI tool that has the potential to
amend individual and team effects of Distributed Asynchronous Team
Environments (DATEs). 2005 Paper No. 2150 |
|
|
Simulation and Network-Centric
Emergency Response
Dennis McGrath, Susan P. McGrath Institute for This paper describes research into the integration of game-based simulations and incident command technologies. The research is at the confluence of two technology revolutions: the influx of new emergency response technologies for emergency responders and the use of computer game technology for serious simulation. Network-centric emergency response is emerging as a new paradigm, driven in large part by technologies derived from the military sector. Specifically, sensor and information sharing technologies are being introduced to civilian incident command technologies and initiating a revolution in emergency response like the “revolution in military affairs” that has been ongoing for the past decade. The implications of these new technologies for well-established incident command procedures are unclear. Emergency responders will soon be capable of generating vast amounts of data. How will operators-in-the-loop (both incident commanders and first responders) deal with this flood of information? The integration of game-based simulations with new and
prototype emergency response information systems may help answer these
questions. Game engines are
particularly well suited for multiplayer, real-time representations of
reality, and game-based emergency response simulations integrated with information
systems can provide a powerful method for evaluating human system
integration. Several pilot efforts
have demonstrated the value of integrating games and prototype emergency
response systems. These include the evaluation and demonstration of mass
casualty monitoring system using a mass casualty simulation, and the
integration of emergency operation center (EOC) software with a radiation
response simulation. We describe these
pilot projects and plans to use simulation to rehearse high-consequence events,
which will not only lead to better emergency response technologies, but will
also allow policymakers to adapt to the new world of network-centric
emergency response. 2005 Paper No. 2244 |
|
|
Current and Future Net-Centric
C3: Usage and Preferences
John S. Barnett and Paula J. Durlach U.S. Army Research Institute Net centric command, control, and communications (C3) is an
important part of current military operations. As the current force moves towards the
future force in the form of Future Combat Systems (FCS), net centric C3 will
become even more valuable. This research sought to extrapolate net centric C3
usage trends from the current to the future force. To do this, two similar studies were
compared. One study asked Soldiers
with Army Battle Command System (ABCS) digital C3 systems experience in
real-world deployed operations about how they employed these systems, and the
second asked similar questions of Soldiers conducting research in an
experimental future force C3 simulator designed to explore FCS concepts.
Participants were given lists of digital C3 functions common to both ABCS and
the future force simulator and asked to rate whether they preferred to
perform the functions digitally or manually, how frequently they used each
function, how difficult it was to learn how to use the function, and how
difficult it was to use the function.
The comparison of these two groups showed there are a number of
similarities between current and extrapolated future usage patterns, but also
some differences. Leaders and Soldiers
in the future force simulator preferred using digital methods to perform more
C3 functions, and said they performed those functions more often than
Soldiers using current ABCS systems.
The research also found a number of C3 functions were easier to learn
and use in the future force simulator versus current ABCS systems. The results suggest digital C3 systems will
be better utilized and more preferred as they become more interoperable. This
knowledge can be used to help further develop future systems and design
future training programs. 2005 Paper No. 2007 |
|
|
Measuring Situation Awareness
for Dismounted Infantry Squads
Donald R. Lampton Army Research Institute Mica R. Endsley SA Technologies Michael T. Gately ScenPro Joseph V. Cohn Naval Research Laboratory Jared Freeman Aptima Glenn A. Martin martin@ist.ucf.edu Situation
Awareness (SA) is a fundamental aspect of contemporary military operations.
Our paper provides the background for and describes the first evaluation in a
planned three-year program to develop new methods of measuring and providing
performance feedback on team SA. Our initial focus is on small teams of
dismounted infantry conducting training exercises using virtual simulations.
A three-pronged approach to measuring SA is planned. The first employs
systematically administered SA probes scored in real-time during the team
exercise. This approach will be augmented by trainer ratings of SA
performance. The second approach involves the capture and analysis of voice
communications among team members. A new system incorporates automated
analysis of voice communications, including communications flow (sender,
receiver, and duration) and content (category of transcribed communication).
The third approach involves monitoring the networked simulation data stream
during an exercise. Predefined rules combine assessments of movement,
orientation, and weapon use to produce measures of decision making and SA.
The three approaches to measuring SA were assessed in the context of an After
Action Review (AAR) for squad-level exercises. Squads conducted urban
missions using the Squad Synthetic Environment at the Fort Benning Soldier Battle Lab. The exercises will employ
scenarios developed specifically to support the measurement of the squad’s
ability to gain, maintain, and apply SA. In future years the SA measures will
be used to provide performance feedback to the trainees during an 2005 Paper No. 2293 |
|
|
Human Factors in Air Force
Flight Mishaps: Implications for
Change
Robert T. Nullmeyer, Air Force Research Laboratory Gregg A. Montijo, Stephen W. Harden Crew Training International Each service expends considerable effort to document human
factors that cause or contribute to flight mishaps. One goal of this activity is to avoid
repeating the same mistakes with the same undesired outcomes. The service safety centers generate
mishap-related reports at several levels of granularity. In the Air Force, mishap summaries are
routinely distributed and reviewed for applicability to academic or simulator
training in accordance with the instruction that specifies Cockpit/Crew
Resource Management (CRM) training.
More detailed information regarding pilot/crew behavior exists in a 2005 Paper No. 2260 |
|
|