Key Concepts

SESAR HP Assessment Process uses: an ‘argument’ and ‘evidence’ approach.

HP argument - the claim that needs to be proven.
Evidence – shall be provided through the HP assessment to show that the HP arguments have been considered and satisfied by the HP assessment process.

It is structured into validation phases that are the set of requirements that the chosen solution has to comply with.
A “solution” is a SESAR specific term which refers to the idea that has to be implemented and which needs to be checked in the analysis. The process produces the HP Log which contains all the data/information obtained from all HP activities conducted as part of the HP assessment.

The table shows an excerpt of the HP argument table, illustrating the first argument and sub-arguments dealing with human roles and responsibilities.

Benefits

SESAR HP Assessment Process:

Provides assurance that HP aspects related to SESAR technical and operational developments are systematically identified and managed;

Describes arguments and necessary evidence to show that airborne and ground ATM actors will contribute to the SESAR expected performance benefits;

Shows that the roles, responsibilities and tasks of airborne and ground ATM actors as developed in SESAR are within the scope of human capabilities and limitations;

Ensures Human Factors proactively contribute to building the operational concept and system architecture.

How It Works

SESAR operational and technical development is structured into three validation phases (V-phases):

V1: Scope – Identifies the operational and technical solutions for meeting the target performance.

V2: Feasibility – Develops and explores the individual concept elements and supporting enablers until:

• the retained concept(s) can be considered operationally feasible.
• further development is no longer justified.

V3: Pre-industrial development and integration – 3 main focuses:

1. Develops and refines operational concepts and supporting enablers to prepare their transition from research to an operational environment.
2. Validates that all developed concepts and supporting enablers can work together and are capable of delivering the required benefits.
3. Establishes that the concurrent packages can be integrated into the target ATM system.

The HP assessment process includes four steps, each of which contains a series of tasks:

STEP 1: Understand the ATM concept.

Understand the ATM system, its needs and constraints (especially the needs of ATM actors), the ATM change proposed by the solution, the assumptions on which the solution is based (i.e. context of implementation);
Ensure that the descriptions of reference, solution(s) and assumptions are complete and consolidated;
Understand potential interactions of the proposed solution(s) with other related SESAR solutions.

STEP 2: Understand the HP implications.

Understand the proposed changes in terms of relevant HP arguments and activities;
Identify HP issues, benefits and impacts associated with the HP arguments;
Define HP validation objectives, expected evidence and relevant scenarios;
Plan the HP activities to be carried out as part of the next step and contribute to SESAR documentation.

STEP 3: Improve and validate the concept.

Conduct the HP activities as they have been specified in the HP assessment plan.
Analyse and document HP activities & outcomes. The findings will: i) serve to support the arguments and validate the proposed concept or design of a technical solution; ii) form the basis of recommendations and requirements for further analysis and development of the proposed solution.
Update HP Log and contribute to SESAR documentation.

STEP 4: Collate findings and conclude on the transition to the next V-phase.

Assess whether HP arguments are satisfied;
Conclude on the transition to the next V-phase;
Produce the HP assessment report;
Manage HP requirements and recommendations.

Illustrative Example

Multi- Sector Planning
Let’s consider the SESAR Solution which investigates the concept of operation and the associated system support required for operating in new team structures such as Multi- Sector Planning.

REFERENCE SCENARIO: The organisation of the controller teams is important to support the safe management of the anticipated growth in traffic and provide ATM in the most efficient manner. Currently the staffing configuration within each sector of En-Route airspace consists of two controllers: i) Planner Controller (PC) – responsible for the planning of flights into and out of the sector, and ii) Executive Controller (EC) – who communicates with the aircraft and achieves the plan ensuring the aircraft maintain separation.

SOLUTION SCENARIO: The traditional Planner-Executive (1P-1E) two-person ATC sector team will be modified with the introduction of a new Planning role: the Multi-Sector Planner (MSP) where the Planning Controller is responsible for the airspace that is under the executive control of two or more independent Executive Controllers (1P-nE).

ANALYSIS: Following the steps we develop the analysis

STEP 1: Understand the ATM concept
1. Description of the reference scenario
2. Description of solution scenario
3. Consolidated list of assumptions
4. List of related SESAR Solutions to be considered in the HP Assessment. The Solutions that have an impact on the HP assessment of the SESAR Solution are the following: i) Extended Arrival Management (AMAN) horizon – ii) Controlled Time of Arrival (CTA) in Medium density / medium complexity environment.
5. Identification of the nature of the change. The table below represents a template which contains the HP Argument Branches. One example of the changes that are expected to occur is given for every such branch.

STEP 2: Understand the HP Implications.
Identification of relevant arguments, HP issues & benefits and HP activities. Some examples of arguments are listed below. The pattern stays the same for the complete list. The complete list of the arguments is found in the HP case.

STEP 3: Improve and Validate the concept.
Description of HP activities conducted. This section provides a summary of activities conducted within the current V-phase to identify HP issues, benefits and impacts. There is one table for each activity conducted. We shall consider one such activity as an example. Next activities shall follow the same pattern.

STEP 4: Collate findings & Conclude on transition to next V-phase
Collect the findings from the HP activities conducted to ensure traceability and decide whether all the required HP activities have been carried out to satisfy the HP arguments for the given V-phase. The table reports in more detail the outcomes for each HP issues/benefit included in the HP assessment plan and the recommendations and/or requirements generated. One issue was selected as an example.

The process is iterative so each step can be done several times through each validation phase.

Key Concepts

The key concept is that when HP is managed properly, system performance will increase. So, organisations have to focus on HP management, not do it ad hoc. The need for a Human Performance Management Standard of Excellence can be summarised as follows: 

• Identify which of the key Human Performance Management areas of focus.Identify where your organisation is doing well and where improvements could help business performance.
• Identify what other similar organisations are doing in this area.
• Identify the level of maturity your organisation needs to get to, considering the size and scale of its operations.
• Identify first or next steps to take in managing human performance.

The HPM SoE describes the levels of maturity in an objective way for the 12 areas of Human Performance.  There are systemic elements like Policy strategy resources, Leadership, Roles and responsibilities and HPM assurance, which indicate that an organisational commitment to HP management is a key issue. The remaining eight elements focus on different operational procedures. 
 

The Human Performance Capability Profiling (HPCP) approach for SAFEMODE

The HPSoE is ATM-focused, and SAFEMODE needed to develop something for the wider aviation system of systems, and also for Maritime. Using the same basic framework, the questions have been adapted slightly in the HPCP to be more generic, this is described further in the section How it Works. 

Benefits

The HPCP approach gives an aviation or maritime organisation a snapshot of its capability in managing human performance, highlighting where it is strong, and where there may be gaps or needed improvements. Critically, HPCP can help organisations see where to place effort on supporting human performance.

As more organisations carry out the HPCP exercise in SAFEMODE, this allows them and others to ‘benchmark’ their own human performance capability against similar companies. 

According to the EUROCONTROL and FAA Action Plan 15 around 70% of total project cost is determined in the first 10% of the project. It is much more cost-effective (60 to 100 times) to change the design of a system in the initial phases of development than to do so once the system has been built and is in operation. HPCP supports human performance implementation in the design phase and in the change process. 

HPCP provides a unified framework for the assessment and management of HP, it can enhance safety and human performance communication, ensuring having the right professional resources in the organisation. 

As a benefit, new HP platforms may emerge in organisations where there is no HP unit, so specific teams of qualified human factors specialists can work together and be systematically integrated into design, selection, training, and safety functions. 

The systematic use of HPCP can help focus attention on Human Performance in planning, design, operations and maintenance, ensuring it is treated as seriously as other business-critical functions.

HPCP can provide objective data on Human Performance Management, ensuring a step-by-step performance improvement opportunity. It fosters benchmarking on HP and planning in the long run as well. 

Applying HPCP can have a balancing power in making the decisions of the company. The principle here is to use multiple lines of evidence that may suggest certain deficiencies in certain processes. Such information will prove invaluable in verifying the results of the assessment, but also point at which elements (if any) need improvement and their priority.

How It Works

The organisation defines the scope of the assessment - only one division or function in the company, or the entire organisation

ANSP selects 1-2 coordinators (safety and human performance but it can vary at ANSPs) who manage the assessment of the 12 areas using the expertise of staff. Coordinators support common understanding of the assessment. 

The involvement of experts from the different fields contribute to the objective data collection that is why it is important to cooperate with key persons having the appropriate knowledge. 

Coordinators collect the justifications for the big picture. 

Each element starts with an objective, which describes what the element is in support of. The five levels of maturity go from A: Initiating through to E: Continuous Improvement. Against each level of maturity there are a series of capability statements, which describe what will be in place once an organisation reaches a particular level of maturity. It is recommended that an organisation go through all the assessment questions up to Level E. Experience has shown that it is possible to meet some higher maturity requirements while other lower maturity requirements are not met. This helps in getting a complete map of which requirements are met for any given element which, in turn, can help in determining which element should be prioritised and any associated actions.

The assessment is not a single activity but a systematic one with the goal of continual monitoring of an organisation’s HPM. Once an organisation has achieved a certain level it does not mean that it stays there. It can move next time to a more mature one if its scale of operations warrants it, or if this is seen as best practice by comparable organisations. HPCP is a means for both assessing the current level of maturity with respect to human performance management and using the result to identify an organisation’s priorities for improvement and the actions that should be undertaken.

Illustrative Example

HPCP Application 
The following is an extract from the HPCP assessment for one division of EUROCONTROL, a pan-European organisation for the safety of air navigation. This particular division is concerned with research and innovation (a separate HPCP exercise was carried out for EUROCONTROL’s air traffic operational centre at Maastricht in the Netherlands. The element assessed below is the first one, Policy, Strategy and Resources, which to a large extent captures how seriously the organisation takes Human Performance. In the assessment the level achieved is assessed as Level 4, even though two aspects of Level 5 are also achieved. This is in part because some other aspects at Level 4 and Level 3 are not fully achieved. The area to improve here would be the two sub-elements at Level 3 which are currently assessed as ‘Y/N’, and to improve matters so these can be clearly answered as ‘Yes’.
 

Key Concepts

The key focus of SHELL Model is on the interactions between its different components: a system requires the contribution of all components to function, and any dysfunctional interaction is a potential source of performance failure. The human element is the most flexible component, capable of compensating for other components’ reduced contributions. SHELL is a useful method for gathering investigation data in a systematic manner. It is used for the initial sorting of facts.

Benefits

It is a systematic approach to analyse and identify problems. It draws attention to the relationship between the human elements and the other factors. In incident investigation, it assists both the investigation and the presentation

of factual information in the report.

SHELL suggests ways in which the probability of an issue may be diminished and appropriate means of detection and correction to be applied. It is a useful model for:

Safety analysis – framework for collecting data about human performance and contributory component mismatches during incident/accident analysis or investigation (e.g., used in SOAM – Systemic Occurrence Analysis Methodology).

Licencing – to help clarify human performance needs, capabilities and limitations thereby enabling competencies to be defined from a safety management perspective.

Training – used to help an organisation improve training interventions and the effectiveness of organisation safeguards against error.

How It Works

SHELL model can be seen from 2 points of view: 

1) The blocks: the SHELL building blocks. Data should be gathered across these areas: 

• Liveware – the human element (personnel).
• Software – procedures, manuals, symbology, etc.
• Hardware – equipment, workplace layout, etc.

2) The interfaces: relationships between each of the components of the model.

The interface between people. In this interface, we are concerned with leadership, co-operation, teamwork and personality.

The relationship between the human and the machine. Includes subjects such as cockpit and workstation configuration, display and control design, seat design and configuration.

The relationship between the individual and the internal and external environments. Includes weather, terrain, physical facilities, infrastructure, economic situation.

The relationship between the individual and supporting systems found in the workplace. It includes regulations, manuals, checklists, publications, standard operating procedures and computer software design.

Illustrative Example

Aviation domain
Runway Overrun

SCENARIO: An aircraft overrun the runway while landing at the airport. The overrun occurred after the aircraft aquaplaned on a runway which was affected by the presence of water following very heavy rain. The aircraft sustained substantial damage. There were no serious injuries reported by the flight crew, cabin crew or passengers. The report of the incident helps to understand the context and the background of the incident. It highlighted a series of errors and non-compliances to the requirements related to the level of training of the crew, the use of the available equipment, gaps in the rules and regulations on top of which were added the contextual conditions (rain, fatigue of the FO).

The table below shows how some of the data collected on the accidents were organized using the SHELL Model. The table actually represented a preliminary step in the process of developing a complete Accident Analysis.

Maritime example 
Propulsion failure in 500m oilrig safety zone

SCENARIO: A Platform Supply Vessel (PSV) has just positioned itself next to an oilrig using the vessel’s Dynamic Positioning (DP) system within the 500 m safety zone around the oil rig. It is 5 am in the morning and still dark outside. The weather is currently picking up with increasing wave height and wind. The DPO (DP Operator) has done a risk assessment evaluating that the operation will be finalized and the vessel can exit the 500 m safety zone before the weather becomes too harsh. Just after connecting hoses hoisted down from the rig crane and the offloading of toxic waste from the rig has started, the DPO experiences unstable conditions in the DP system. The DP system gives many alerts concerning loss of position and also some unstable feedback concerning the propulsion system. The DPO tries several different approaches to regain control of the system, but realizes he/she has lost control of the vessel and it is drifting towards the platform legs. The vessel hit one of the platform legs and its underbody approximately 30 minutes after the problems started. The vessel’s bridge crew tried to regain control, but the vessel and the platform sustained medium damages. The production on the oilrig was stopped and the rig crew evacuated according to standard procedures. 

The report of the incident helps to understand the context and the background of the incident. It highlighted a series of errors and non-compliances to the requirements related to the level of training of the crew, the use of the available equipment, gaps in the rules and regulations on top of which were added the contextual conditions (rain, waves, bridge crew shifts etc.).

The table below shows how some of the data collected on the accidents were organized using the SHELL Model. The table actually represented a preliminary step in the process of developing a complete Accident Analysis.

Key Concepts

Fatigue arises from the combination of four ‘fatigue factors’:

Cumulative fatigue: Cumulative fatigue refers to the “amount and quality of sleep attained during a shift cycle”. If you sleep more and it is quality sleep you are less fatigued.  If you sleep more but it is poor quality sleep (broken or disturbed) then you still feel fatigued.  Good sleep is the only way to address the sleep imbalance that leads to cumulative fatigue. Napping and taking short rests doesn’t help with cumulative fatigue.

Position in the shift cycle: Position in the shift is determined by three sub factors: the start time of the shift, the amount of time on duty, and the position in the natural circadian rhythm.  These three factors combine to produce an impact on fatigue that relates to the impact of the shift itself.  Further into the shift cycle, and the daily shift point contribute directly to the experience of fatigue.

Workload: Operators with high workload will experience more fatigue.  The workload contribution to fatigue however only affects overall fatigue after periods of work longer that 120min.  Within many European aviation operations, extended periods of high workload are mitigated through rest periods and periods of lower workload during operations.  This is not necessarily the case in maritime.

Recovery: The final component is that of recovery, comprising breaks and naps.  The impacts of breaks and naps reduce the impact of fatigue and help to ‘refresh’ the operator but only temporarily.  Breaks and naps only alleviate the fatigue induced by impact of the position in the shift cycle. 

Fatigue Risk Management: Understanding and assessing these four contributors to fatigue is a key component in any fatigue risk management system.  But if we are managing the impacts of fatigue, then realistically, it’s a little late.  The real key is to develop a system where an understanding of how fatigue is generated (by the operation) is embedded in the design of the system not considering only the HMI, but also staffing, and shift working as part of the concept of operation.

The figure below (developed in the aviation domain, but generally applicable) shows a more exhaustive list of factors impacting fatigue and human performance, some of which are easier to manage than others. In particular, some factors are related to life outside work, and individual differences in how quickly we are each fatigued, and for how long. This figure highlights the challenges of managing fatigue.

Benefits

Fatigue is cited as a contributor to operational incidents in the aviation and maritime industries, and there are limited tools available for operators to assess fatigue other than self-reporting (which has been shown to be unreliable).  

Fatigue risk management systems are effectively mandated by regulation. 

Understanding the extent to which operators are impacted by fatigue and the extent to which each of the four components can contribute to the fatigue of the operators means that managing the impacts of fatigue is more easily targeted.  This makes the fatigue risk management system more effective and makes for greater individual efficiency as the impact of fatigue is minimised.

Current estimates (academic sources) show that fatigue is cited in over 20% of operational incidents, specifically in air traffic, fatigue leading to low vigilance has been reported by 80% of controllers.  Quoting from the Maritime Injury guide for 2014 - “…seamen on two vessels with watchkeepers may end up working 84 or more hours per week and are afflicted by chronic fatigue. This makes officers and crew prone to make errors in judgment, react slowly in emergency situations, and suffer injuries in on-board accidents that otherwise are avoidable…”

It is clear that fatigue needs to be addressed, not just in operations, but also in the design and management of the system and during the operation within which we expect people to function for operations extending over 24h.

How It Works

Examples of positive effects on fatigue risk management include:

• Design work (operator tasks) that assure alertness by providing additional tasks during periods of low workload.
• Design equipment that the operator uses (chairs, workstations etc.) to provide an alarm or an alert after periods of inactivity (i.e. when the operator may have fallen asleep)
• Consider fatigue as a contributor in the incident reporting system
• Provide the means by which colleagues can cross report where co-workers appear to be suffering from fatigue.
• Embed automated detection technology that allows fatigue to be estimated during operations: monitoring of eye blink, head movements etc.
• Ensure an open culture of reporting that facilitates self-reporting of fatigue without punitive measures being taken.
• Deploy a method of assessing risk arising from Fatigue – figure summarised from IATA Fatigue Risk Management Systems (FRMS) 2011: 

The figure above is the basis of what is known as a Fatigue Risk Management System (FRMS), which is used to manage the risks arising from fatigue (An FRMS is often incorporated into the organisation’s Safety Management System). Reviews of incidents, as well as surveys are used to identify where fatigue can cause safety risks, and counter-measures can then be employed in those areas. However, because fatigue can vary due to the factors described above, there needs to be monitoring of risks associated with fatigue. The approach of identifying, measuring and monitoring becomes an iterative adaptive cycle, aimed at progressively minimising or even eradicating adverse effects linked to fatigue.

The figure below to shows some of the types of tools that exist to support the management of fatigue and also the development of a FRMS. Bio-mathematical models aim to predict fatigue (e.g. for considering shift and roster arrangements), and work with varying success. Actigraphy aims to measure sleep and sleep quality. Fatigue reports, sleep logs and subjective evaluations (rating scales) and web-based surveys can be used to assess subjective feelings of fatigue, and expert reviews and observation/measurement of performance and performance decrement can also inform a better understanding of fatigue in normal and abnormal work conditions.

Illustrative Example

Aviation Domain: Air Traffic Control
In 2011 an ‘infamous’ incident arose in which two aircraft attempting to contact air traffic control at Washington’s Ronald Reagan airport were unable to raise the ATC on the radio telephone.  Despite repeated calls to the tower, two pilots had to affect landings normally intended for unmanned airfields.  The incident was widely reported and investigated and led to the suspension of the controller on duty at the time.  Typically, the controller was blamed for falling asleep while on duty.

There was more to the incident than a controller falling asleep on duty.  The following contributors to the event existed but were not widely reported:

• Because traffic was low during the night shift management reduced staffing to one controller for night shift.
• Only one controller was on duty, so there was no back up or relief between 22:00 and 06:00.  
• The controller had only had 10h of rest between successive shifts.
• Because there was only one controller on duty there were no rest breaks scheduled.  
• Napping as a means of recovery from fatigue was not allowed.
• The controller was working his/her fourth consecutive night duty (and still had 1 more shift remaining).
• There was a history of controllers falling asleep when working the night shift that were not widely reported in the press.
• There was no means of recording fatigue as a contributor to incidents in the FAA incident reporting system

The incident ultimately evoked a response from President Barak Obama and the following quote: 

 “What we also have to look at is air traffic control systems. "Do we have enough backup, do we have enough people, are they getting enough rest time?

 How could the system design have been managed to ensure the risk arising from controller fatigue were minimised?

Maritime Domain: Collision Accident
May 21st, 2018 the Master instructs that the Officer Of The Watch (OOW) is the sole watch-keeper at 03.30.  OOW was feeling tired and engaged in some brisk walking around the bridge to try to wake up a little.  Felling a little more refreshed the OOW then took a seat at the radar.  At 04.40 the OOW is woken up abruptly as the ship hits a sea wall of a nearby bridge at 15kts.
The collision occurred because the OOW was so tired he fell asleep while in command of the ship.  Several factors contributed to this situation.  It was not just a case of poor seamanship:

• The OOW did not feel able to report his fatigue to the master of the vessel.
• The ship was not equipped with a bridge watch navigation alarm system.
• No lookout was assigned to the OOW, interaction with the lookout should have kept the OOW awake.
• The OOW had not had proper sleep for 4 days 
• The design of the bridge encouraged sitting which meant it was easier to fall asleep
• The use of the autopilot meant the OOW had few tasks other than monitoring its performance
• Routine absence of a lookout during night time operations creates the impression that night operations are safe and require less monitoring.
• Being in good visibility creates the impression that night time operations are safe AND the master of the ship allows reduced manning.
• A three-watch system was operating, but without an adequate level of staffing available for the bridge.

The culture of operations at night with good visibility, and the sense that the OOW could not report his fatigue combined, on the night, with bridge design and equipment availability to result in the collision.

Key Concepts

The LOAT brings into discussion important matters related to systems’ innovation.

Automation is not only a matter of either automating a task entirely or not but deciding on the extent to which it should be automated.

Introducing automation brings qualitative shifts in the way people practice rather than mere substitutions of pre-existing human tasks.

Classification of different levels of automation, recognises various ways in which human performance can be supported by automation.

The “optimal” level of automation in a specific task context is about matching the automation capabilities to a number of operational situations, while increasing the overall performance in efficient human-machine cooperation.

It provides potential alternatives for designing the human-machine interaction by taking full benefit of the technical solution.

Benefits

The LOAT can be used to:

Compare different design options and determine the optimal automation level in all operational contexts. The best level of automation depends on the capabilities of the automation itself and on the operational context in which it is included. You can use the LOAT to orient yourself in these choices.

Understand the impact of automation, both positive and negative, on the system. The highest level of automation is not always the best one. Finding the right level helps to make sure that you are taking the full benefit from it, without being negatively affected by side effects, such as nuisance alerts or erroneous directions.

Get insight on how to prevent human performance issues and take full benefit of available technical solutions in various domains. A common taxonomy for different domains will mean a shared approach to automation design. Developed in the Air Traffic Management domain but applicable to all domains in which automation plays an important role in sustaining human performance.

Classify existing implementations and derive lessons to improve automation design. Classifies the ‘nuances’ of automation enlightening the possibilities in design. Modifying existing tasks or introducing new ones involve the use of different psychomotor and cognitive functions, which implies the adoption of different automation solutions.

How It Works

The LOAT is organized as a matrix:

I. in horizontal direction: the 4 generic functions, derived from a four-stage model of the way humans process information:
1) Information acquisition: Acquisition and registration of multiple sources of information. Initial pre-processing of data prior to full perception and selective attention;
2) Information analysis: Conscious perception, manipulation of information in working memory. Cognitive operations including rehearsal, integration, inference;
3) Decision and action selection: Selection among decision alternatives, based on previous information analysis. Deciding on a particular (‘optimal’) option or strategy;
4) Action implementation: Implementation of a response or action consistent with the decision previously made. Carrying out the chosen option.

II. in vertical direction: each cognitive function groups several automation levels (between 5 and 8). All automation levels start with a default level ‘0’ - corresponding to manual task accomplishment - and increase to full automation.
 

The initial methodology that has been defined for the LOAT incorporates 4 steps: 1) Identify the automated tool to be classified; 2) Determine which function is supported (can be more than one); 3) Determine the relevant cluster of automation levels; 4) Identify the specific automation level and consider the design principles associated to it. The main LOAT goals are:

• it is a method by which we can efficiently identify the most suitable human-automation design;
• based on the classification of already developed automation examples we can provide specific and relevant human factor recommendations.

Illustrative Example

Aviation Domain
To see how the LOAT can be implemented in practice we should consider an example of human-machine interaction in industry.

SCENARIO: The Remote (or virtual) Tower uses an array of sensing and communication technologies on the airfield and the surrounding airspace, while providing the actual control functions (using certified controllers). In matter of automation the remote towers come with the following technology:

• surveillance is generally provided by an array of video cameras, both conventional and infrared, to replace and improve upon the controller’s OTW view.

• communications between controllers at the RTC and aircraft that use the airport are primarily via radio.

• displays typically include an “out-the window” display of the airfield, data from radar (if available) and any other surveillance method. They may also include equipment indicating weather conditions (such as a wind rose), a compass rose, and geographical overlays.

• controls would include microphones lighting controls, camera controls, etc.

• other technologies include object tracking and alerting (in the air and on the ground) and features such as automatic runway intrusion alerts.

As in a conventional tower, all surveillance and communications information is transmitted to one or more controller positions in the Remote Tower Center (RTC).

ANALYSIS: Following the aforementioned steps, we can use the LOAT to identify the level of automation.

• Step 1: Identify the automated tool to be classified. From the technologies of the RTC mentioned in the SCENARIO we choose the one we want to classify: e.g. Surveillance

• Step 2: Determine which function is supported. We should then consider the main purposes of surveillance cameras: i.e. they collect images. Considering this, the function supported is: Information Acquisition (A).

• Step 3: Determine the relevant cluster of automation levels. While the cameras are the ones collecting and generating the image to the user but only on the user’s demand, the relevant automation level is: supported by Automation. 

• Step 4: Identify the specific automation level. The user is still in control to decide whether the information received from the cameras is relevant or not, and can filter the information received, so the automation level is: with user filtering and highlighting of relevant info (A2) 

In this way the LOAT can be used to decide upon the optimal level of automation suitable for the user and can help to establish whether or not it is a good choice to automate the system in the first place.

Key Concepts

• Human HAZOP is a Group method, rather than a single-analyst approach such as TRACER or STAMP.

• It relies on ‘structured brainstorming’ by a small group (e.g. 6-8) design, operational and Human Factors experts, led by a HAZOP chairman

• It requires a representation of the system, procedure or interface being evaluated. Typically design documents and drawings (or photos if operational) are present, along with a task analysis 
   (typically Hierarchical Task Analysis or Tabular Task Analysis which details how the human operators are intended to interact with the system.

• The HAZOP study group proceeds through each step of the procedure or task analysis and considers potential deviations from the expected behaviour, prompted by HAZOP guidewords.

• The consequences of deviations from the intended functioning of the system are considered, as well as existing safeguards & recovery means

• Required safety recommendations are stated and recorded by the HAZOP secretary.

• HAZOPs are formally recorded and logged.

Benefits

HAZOP has survived a long time, and has a long list of benefits:

• Human HAZOP provides systematic and exhaustive design review, and can lead to the discovery of new hazards.

• It requires limited technical training – it is an ‘intuitive’ method.

• The use of a team gives a range of viewpoints.

• It has a good track record in several industries.

• It is versatile, and can be applied to all sorts of design formats.

• It can be used early on (e.g. at the concept design stage).

• It is good at finding credible vulnerabilities, including violations.

• It has high acceptance by both designers and operators, as they both input to the HAZOP.

• It creates a good ‘bridge’ between design and operations.

On the negative side, there are several well-known disadvantages:

• HAZOP, especially if applied to a complete system, is resource intensive.

• The GIGO (garbage in, garbage out) principle applies. If, for example, inexperienced operational people are used as experts, don’t expect too much.

• HAZOP tends to concentrate on single deviations or failures – it does not look at how different failures might interact – only risk modelling does this.

How It Works

Human HAZOP works by applying a set of guidewords to the chosen design representation, be it design drawings and documentation, task analyses, or more dynamic visualisations (e.g. working prototypes) of the system under investigation.

The guidewords are as follows shown in the table below. As an example, the most commonly used guideword is ‘No’, meaning the human operator does not do something when required (e.g. failing to lower landing gear). Guidewords such as More or Less could be applied to entering correct weight into software used to manage balance (aviation) or ballast (maritime). Reverse is rarer but can still happen, as in the AF447 air crash where the co-pilot manoeuvred the controls in the wrong direction, possibly due to startle response and inexperience, putting the plane into an irrecoverable stall. In maritime it could mean setting thrusters in the wrong direction when docking. Early and Late could relate to use of flaps or deciding when to descend an aircraft, or when to turn to pass another vessel (maritime). The most interesting guideword is ‘Other than’, wherein a completely unintended action is the result. This requires operational experience rather than fanciful imagination, and typically in a HAZOP only one or two (or none) such possibilities are identified. 

In the HAZOP session, the HAZOP Chairman leads the group through each step of the activity associated with the interface or procedure, and each guideword is considered in turn. The table below shows the possible guid words and their definitions.

The results of a HAZOP are logged in a HAZOP table by the HAZOP secretary. Mitigations may not always be apparent, in which case actions are given to HAZOP team members or their departments to develop mitigations and report back to the group or design authority to ensure they are sufficient and mitigate the hazard. The table below shows a generic HAZOP table including indications for the content to fill in.

Illustrative Example

Aviation domain
The new Air Traffic Controller systems shifted the focus away from one primary form of information presentation and usage (paper) to a completely different presentation medium (computer screen).  Additionally, although both these media were visual in nature, the computerisation of flight progress strips paved the way for automated ‘up-linking’ of messages to the cockpit of aircraft, reducing the need for oral communication for a number of tasks. This transmission of information is known as ‘data-link’. Given the high performance and proven nature of the pre-existing system, it was considered appropriate to thoroughly evaluate the transition to the new system interface.  A HAZOP analysis was therefore conducted on the proposed HMI (Human Machine Interface) of the (electronic) flight progress information system.

The HAZOP team that assessed the implications of the new interface was made up of three designers, one air traffic controller and two human factors specialists. The study identified a number of ‘vulnerabilities’ in the prototype system and ‘opportunities’ for error that needed to be designed out or worked around (e.g. via procedures and training).  A total of 87 recommendations, covering changes to the HMI, improved feedback and training / procedure improvements were generated in the three HAZOP sessions.  

The HAZOP approach proved surprisingly useful and productive of agreed changes in interface design. Moreover, since the designers were not only present, but were actively involved, any design changes they thought necessary were simultaneously accepted for implementation. The HAZOP therefore had very fast and effective impact on the design process. To illustrate the meeting output, part of the study is shown in the table below. 

Maritime domain

In shipping, the techniques to represent the geophysical characteristics of the seabed have shifted the focus away from paper-based nautical charts to computerized electronic charts such as sonar charts. A sonar chart is an HD bathymetry map featuring extraordinary bottom contour detail for marine and lakes, excellent for increasing awareness of shallow waters and for locating fishing areas at any depth level.

Although both techniques are visual in nature, by computerising the seabed mapping, it is possible to increase the visualization details and to reduce the need for communication and coordination amongst crew members. Therefore, a Human HAZOP analysis can be conducted on the proposed human-machine interface of the new electronic system. The outcomes of this study were used to identify a number of ‘vulnerabilities’ in the proposed system and ‘opportunities’ for error that needed to be designed out or worked around (e.g. via training in the use of sonar chart).To illustrate the HAZOP output, an example is shown in the table below.

Key Concepts

The TRACEr technique is based on the human information processing paradigm which claims that the human mind is like a computer or information processor. It contains eight interrelated taxonomies that altogether describe:

The context of the incident – using:
1. Task Error – describing controller errors in terms of the task that was not performed satisfactorily.
2. Information – describing the subject matter or topic of the error.
3. Performance Shaping Factors (PSF) – classifying factors that have influenced/ could influence the controller’s performance, aggravating the occurrence of errors or assisting error recovery.

The cognitive background of the production of an error – using:
4. External Error Modes (EEMs) – classifying the external and observable manifestation of the actual or potential error.
5. Internal Error Modes (IEMs) – linked specifically to the functions of the cognitive domains and describing which of the cognitive functions failed/ could fail and in what way.
6. Psychological Error Mechanisms (PEMs) – describing the psychological nature of the IEMs within each cognitive domain.
7. Error detection – describing the error using specific keywords.

The incident recovery – using:
8. Error correction – describing how the error was mitigated.

Benefits

• The TRACEr technique provides feedback on organisational performance before and after unwanted events. 
   Its main strengths are comprehensiveness, structure, acceptability of results and usability.
• It requires moderate resources.
• It offers a possibility to derive error reduction measures.
• It helps to determine what errors could occur, their causes and their relative likelihood of recovery.
• It facilitates the identification and classification of human errors in relation to Human-Machine Interaction (HMI).

How It Works

TRACEr can be used in two different ways, for a retrospective analysis (i.e. following an incident or accident) or for a predictive analysis, in order to anticipate the possible errors, in the context of a risk analysis. For the sake of brevity only the retrospective use is here illustrated, but in two different instances: first the original one, developed in the air traffic control domain, then a modified one adapted to the maritime domain.

Aviation Application
The process for applying TRACER retrospectively in the ATC domain can be represented as a flowchart, and identifying six different steps.

• Analyse incident into 'error events', identifying the task steps in which an error was produced. 
• Task Error Classification, classifying the error with the Task Error taxonomy 
• Internal Error Modes (IEMs) Classification, deciding which cognitive function failed.
• Psychological Error Mechanisms (PEMs) Classification, identifying the psychological cause.
• Performance Shaping Factors (PSFs) Classification, selecting any PSFs related to error under analysis.
• Error detection and Error correction, identifying errors/ corrective actions by answering to four questions.

Once the analyst has completed step number six, the next error should be analysed. If there are no more “error events” then the analysis is finished.

Maritime Application
The TRACEr methodology was adapted to the maritime sector by the World Maritime University and named TRACEr-Mar, it consists of nine coding steps or classification schemes that can be divided into three main groups, which describe the context of the incident, the operator context and the recovery from the incident. The overall process can be summarized with a flowchart and table below provides a brief summary of the nine TRACEr steps.

Illustrative Example

Aviation domain

To see how the technique can be implemented in practice we may consider the following ATCO-Pilot communication scenario which resulted in a loss of separation minima.

SCENARIO:  
A very busy day, young and unexperienced ATCO, tired because of lack of sleep. ATCO issuing clearance for aircraft A: “ABC123 descend FL240”; Pilot A readback: “ABC123 descend FL220”; ATCO issuing clearance for aircraft B: “XYZ789 climb FL230”; Pilot B readback: “XYZ789 climb FL230”.

STCA starts (Short Term Conflict Alert – ground-based safety net intended to assist the controller in preventing collision between aircraft). ATCO: “ABC123 avoiding action turn right.”

A representation of this scenario is illustrated in the figure below.

ANALYSIS focused on the first 'error event' (ATCO’s error): 

Step 1: Develop the sequence of events and identify error events.
. Controller fails to notice error
. Controller is late to respond to STCA
. Controller gives indication in wrong direction

Step 2: Task Error classification 
. Controller Pilot communication error

Step 3: IEM classification
. Readback error

Step 4: PEM classification
. Expectation bias

Step 5: PSF classification
. Traffic complexity

Step 6: Error detection and Error correction

Maritime domain 

SCENARIO: On December 2012, the dry cargo vessel Beaumont ran aground on Cabo Negro on the north Spanish coast while on passage from Coruna to Aviles. At the time of the grounding she was proceeding at full speed, and the Officer Of the Watch (OOW) was asleep. An inspection of the vessel’s internal compartment quickly established that, despite being driven hard aground on a rocky ledge, there was no breach of the hull. The MAIB investigation identified that the OOW had fallen asleep soon after sending his night lookout off the bridge. Available bridge resources that could have alerted the crew and/or awoken a sleeping OOW were not used resulting in Beaumont steaming at 11.5 knots with no one in control on the bridge for over an hour. 

The table below presents an example of applying the TRACEr-MAR taxonomy to the task error: Officer Of the Watch fell asleep on the bridge.

Key Concepts

• HEART method is based upon the principle that every time a task is performed there is a possibility of failure and that the probability of this is affected by one or more Error Producing Conditions (EPCs) (e.g. distraction, tiredness, etc.) to varying degrees.

• Within HEART, those factors which have a significant effect on performance are of greatest interest. These conditions can then be applied to a "best-case-scenario" estimate of the failure probability under ideal conditions to then obtain a final error chance.

• By forcing consideration of the EPCs potentially affecting a given procedure, HEART also has the indirect effect of providing a range of suggestions as to how the reliability may therefore be improved and hence minimising risk.

Benefits

On the positive side, HEART presents a long list of benefits:

• HEART is rapid and straightforward to use.
• HEART has a small demand for resource usage.  
• It has a good track record in several industries such as nuclear, healthcare, rail, maritime, offshore and aviation industries. 
• It can be used early on (i.e. at the development stage).
• Several validation studies have been conducted and concluded that this technique produce estimations that are equivalent to those obtained by applying more complicated methods.

On the negative side, HEART also presents several well-known disadvantages:

• The EPC data has never been fully released and it is therefore not possible to fully review the validity of it.
• HEART relies to a high extent on expert opinion, first in the point probabilities of human error, and also in the assessed proportion of EPC effect. The final Human Error Probabilities are therefore very sensitive.

How It Works

HEART is an Additive Factors Model (AFM) that proposes the application of a multiplicative impact on human reliability calculations. This technique works under the assumption that any estimated reliability of task performance may be modified according to identified EPCs.

The HEART methodology generally consists of two assessment process steps (a qualitative and a quantitative step).

• The qualitative method is started by classifying the generic task based on the accidents data report.  After finding the general task, it then breaks it down to smaller parts called Error‐producing conditions (EPCs). 

• Once this task description has been constructed, the second step of this methodology is the quantitative method, wherein Human Error Probability (HEP) is calculated , usually by consulting local experts. To obtain the HEP using the HEART methodology, first it is necessary to obtain the nominal human unreliability, which belongs to the generic task. When the EPCs are determined, the multiplication number of each EPC is obtained.

Illustrative Example

Aviation domain
This example is derived from an application of CARA (Controller Action Reliability Assessment), an adaptation of HEART specific for the Air Traffic Control domain). In this case CARA is used to asses human performance for anticipated landing clearances during low visibility.

During periods of low visibility, for aircraft landing at airports, a support system was being designed to enable controllers to be provided with information that runways were clear and available for aircraft to land on when they may not be visible from the tower. One key area around the runway in this process is the ‘Obstacle Free Zone’ (OFZ) for which consideration was being given as to the form of alert a controller would receive if the following hazard occurred “The aircraft which landed and is moving clear of the runway stops beyond the trigger line (which enables the next aircraft to start to approach the runway) but not clear of the OFZ with the landing aircraft less than 200 feet above threshold.”

Simple event trees were developed and focused on human error probabilities of two key ATC tasks:

1. Controller identifies the aircraft which has landed is stationary within the OFZ.
2. A warning is issued to the landing aircraft that the aircraft is stationary in the OFZ.

To populate the event tree, CARA values were calculated for these two tasks. The values were produced by two human factors experts and drew on human reliability analysis and event trees from previous hazard analysis work. For task 1, three potential designs were considered and values calculated separately:

1. Use of an audible and visual alert
2. Use of a visual alert only
3. Use of no alert, and relying on controller scanning patterns

Examples of calculations of the human reliability are presented below.

Issue warning to aircraft 2 (Aircraft approaching runway)
Task Description and Assumptions: having identified that aircraft one (the aircraft already landed) is stationary, the controller must identify and warn aircraft 2. The error under consideration is that the controller does not warn aircraft 2, having identified that AC 1 is stationary.

Results: Calculation  = 0.003 x [(11-1) x 0.2 + 1] = 0.003 x 3 = 0.009

The analysis provided the following quantified outcomes for the Hazard using the event trees for the three design scenarios:

• Use of an audible and visual alert - 9.4E-03
• Use of a visual alert only - 1.4E-01
• Use of no alert, and relying on controller scanning patterns 3.1E-01

The analysis identified that much better reliability would be achieved using option A, a visual and audible alert, as there is not a reliance on the controller to scan the display to identify the hazard. It was beyond the scope of this work to validate and take these data forward in to the system design.

Maritime domain
In the illustrative example, the case of ship collision during passage through the waterway (narrow canal) is applied. Human based factors are analysed to systematically predict human error for each task. The collision occurred between a tanker ship and a livestock ship during the narrow canal passage. The tanker ship which were steering with 11,7 knot speed to the course of 244° in Marmara Sea in the direction of Dardanelles, have contacted with her starboard bow to port stern quarter of the livestock ship within 20 nm range of Dardanelles at 06:25 LT on the date of 01.10.2013. Development of the event was as follows; the tanker ship overtook the bulk carrier ship when she was steering close to the middle of the separation in the direction of Dardanelles. It has been observed that livestock ship was steering in the starboard bow of bulk carrier ship in direction of Dardanelles and close to outmost line of the separation where she was also in the range of about 0,6 NM with the tanker ship. It also has been observed that another ship (container ship) was steering in the starboard bow of the tanker ship within the range of 1 mile. 

At the result of investigation of ECDIS records and statements of the officer on watch, it has been perceived that; After taking over bulk carrier ship with the range of 0,3 NM and heading towards to container ship, the tanker ship has contacted with livestock ship which were steering outmost line of the separation. Even though small course alteration to port side that was made by the tanker ship, contact couldn’t be prevented due to sudden and big course alteration to the port side had been made by the livestock ship. The root cause is attributed to the mental fatigue, inadequate manning on the bridge, inadequate standards, lack of communication, poor risk evaluation, delayed collision avoidance manoeuvring, limited time, lack of knowledge and inadequate checking. 

The VTS (Vessel traffic services) was informed about the collision. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer (OOW) was keeping navigational watch at the time of collision. The Master of ship was informed just before of the collision. Helmsman was called by the Master before collision. The hierarchical task analysis was performed to understand relative activities/tasks during ship strait passage. Then, HEART is applied for quantification of human errors. Following table shows details including tasks, responsible person, GTT, EPC, APOE and calculated HEP for the case of ship collision during the narrow canal passage.

Key Concepts

The key concepts of the techniques are:

• Task analysis: To define the activities to achieve the goal. 

• Common Performance Conditions (CPC): Human cognition and action context are determined in accordance with CPCs which provide a well-structured and comprehensive basis for characterizing the circumstances under which performance is expected to occur.

• Control modes: Scrambled, Opportunistic, Tactical, Strategic modes. The control modes are linked with different failure probability intervals representing human action failure probabilities.

• Cognitive activities: These are relevant for work in process control applications and also provide a pragmatic definition for each. Communicate, compare, diagnose, execute, monitor, observe, etc. are some of the cognitive activities. 

• Context Influence Index (CII): This value can be calculated by deducting the number of “reduced” CPCs (i.e. CPCs likely to have a negative impact on performance in the situation) from improved CPCs.

• Performance Influence Index (PII): Specify proper weighting factors for cognitive functions such as execution, planning, observation, and interpretation.

• Cognitive failure probability (CFP):  This is probability of failure for each cognitive failure type.

The main focus of CREAM technique is based on 2 elements:

• human performance - how a system behaves over time and 
• cognition activity description of cognition activity in the system/process.

Benefits

The technique presents a quantitative approach to systematically predict human error for designated tasks and ascertain the desired safety control level. It offers a clear and systematic approach to quantification. The quantification problem is, in fact, considerably simplified because CREAM is focused on the level of the situation or working conditions rather than on the level of individual actions. 

The technique is based on a fundamental distinction between competence and control. A classification scheme clearly separates genotypes (causes) and phenotypes (manifestations), and furthermore proposes a non-hierarchical organisation of categories linked by means of the sub-categories.

How It Works

The technique includes both basic and extended versions. 

The purpose of the basic method is to produce an overall assessment of the performance reliability that may be expected for a task. The basic one consists of the following steps:

1. Scenario development: This step provides a variety of scenarios including time availability, condition of the work place, the environment, stress, noise level, collaboration of the crew, etc. The CPCs are determined according to the scenario.

2. Task analysis of the process/work: The objective of this step is to identify the tasks and sub-tasks of the process according to the hierarchical task analysis (HTA).

3. Determine CPC: These are used to characterise the overall nature of the task. The combined CPC scores are needed to figure out human performance. There are nine CPCs defined in the CREAM: Adequacy of organisation, working condition, adequacy of the man-machine interface, availability of procedures/plans, number of simultaneous goals, available time, time of day, adequacy of training and experience, quality of crew collaboration. For each task, a CPC level is determined.

4. Determine control mode: The control mode is determined from the combined CPC score. The control mode corresponds to a region or interval of action failure probability.

The purpose of the extended version of CREAM is to produce specific action failure probabilities. The following steps are used:

1. Scenario development

2. Task analysis of the process/work

3. Determine Common Performance Conditions (CPC)

4. Identify context influence index (CII): The CII is used to quantify CREAM, in particular CPCs, in order to simplify calculation. This value can be calculated by deducting the number of reduced CPCs from improved CPCs (Akyuz, 2015; He et al., 2008).

5. Determine performance influence index (PII): This step provides PII values which were generated to specify proper weighting factors for entire cognitive functions such as execution, planning, observation, and interpretation. Each CPC has a different PII value which means there are different weighing factors (He et al., 2008)

6. Calculate cognitive failure probability (CFP): The CFP specifies human failure probabilities for each cognitive failure type in order to calculate HEP value. Once the nominal cognitive failure probability (CFP0) has been nominated for each sub-task, the CFP value (HEP) can be calculated respectively (Akyuz, 2015). The following equation is used.

Illustrative Example

Maritime domain
In the illustrative example, the case of ship collision during passage through a strait is handled. Human factors are analysed to systematically predict human error for each task. The extended version of CREAM technique is used.  

Step 1 Scenario development: 
The collision occurred between a tanker ship and a livestock ship during the Dardanelle Strait passage at sea. The collision occurred during overtaking. The VTS (Vessel traffic services) was informed about the collision since it occurred in the Dardanelle Strait passage. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer was keeping navigational watch at the time of collision. 

Step 2 Task analysis: 
Hierarchical task analysis was created covering potential activities/works during ship strait passage.   

Step 3 Determine CPC: 
Assessor determined appropriate CPCs for each task according to the scenario and crew condition.

Step 4 Identify context influence index (CII): 
Assessor determined CII values for each task

Step 5 Determine performance influence index (PII): 
Assessor determined PII values for each task. 

Step 6: Equation was used to calculate the human error probability (CFP) 
for each task.

Following table shows task analysis including responsible person, relevant cognitive activity, cognitive function and calculated CFP for the case of ship collision during the strait passage (He at al., 2008; Apostolakis et al., 1988).

Step 7: Identify significant human error: 
Analyse which of the sub-task of the ship strait passage need further details and if there is any control measure to mitigate human error. 

In the view of analysis, sub-task 2.2 (Proceed inside the separation line) has the highest cognitive error rate where OOW (Officer on Watch) and helmsman failed to observe monitoring ship position. Likewise, sub-task 3.3 (Increase ship manoeuvring ability by consuming diesel oil and steering engines) is another critical activity where human error increased. The Master failed to enhance the ability of ship manoeuvring by consuming fuel oil (instead of diesel oil) during strait passage.  

Key Concepts

People need to coordinate information sources, to compare, copy, and integrate data from multiple applications. Scenarios highlight goals suggested by the appearance and behaviour of the system, what people try to do with the system, what procedures are adopted, not adopted, carried out successfully or erroneously, and what interpretations people make of what happens to them.

Scenarios include a specific setting with agents or actors: it is typical of human activities to include several to many agents. Each agent or actor typically has goals or objectives. These are changes that the agent wishes to achieve in the circumstances of the setting. Every scenario involves at least one agent and at least one goal. When more than one agent or goal is involved, they may be differentially prominent in the scenario. Often one goal is the defining goal of a scenario, the answer to the question “why did this story happen?”

Scenarios have a plot; they include sequences of actions and events, things that actors do, things that happen to them, changes in the circumstances of the setting, and so forth. Particular actions and events can facilitate, obstruct, or be irrelevant to given goals. For example, in an office application resizing a spreadsheet and moving it out of the display are actions that facilitate the goal of opening a folder. While resizing and repositioning a memo are actions that facilitate the goal of displaying the memo so that it can be carefully examined. 

Benefits

• Vivid descriptions of end-user experiences evoke reflection about design issues;
• Scenarios concretely fix an interpretation and a solution, but are open-ended and could be easily revised;
• Scenarios help to identify specific requirements to make sure the envisaged solution will actually sustain the users in performing their tasks;
• Scenarios anchor design discussion in work, supporting participation among relevant stakeholders and appropriate design outcomes;
• Scenarios can be written at multiple levels, from many perspectives, and for many purposes.

How It Works

Scenarios can be developed following a process like the one described below:

1. Draft a narrative description of the scenario starting from your own experience of the activity under analysis or imagining how the future activity will work;

2. Involve other relevant people in the revision of the scenario, e.g. designers, engineers, human factors experts, operations manager, product managers, future end users, either individually or in a dedicated session with all the representatives;

3. Explain to everyone the objectives of the scenario and provide context to make it as accurate as possible, i.e. the who, what, when, where and why detail;

4. Collect data, information, critics, suggestion and ideas from each involved people. Try to understand what are the strengths and weaknesses of the system, what should be removed, what maintained and what improved;

5. If possible, organize a role-playing or simulation to test the scenario and see which are the elements that work properly and those that would require a revision;

6. If needed rewrite or adjust the scenarios by means of different iterations, taking into account the result of the simulation.

Illustrative Example

A simulated futuristic decision support tool for ATM operations

Context: Futuristic scenario in which an adaptive automation functionality in integrated into the ATM system, capable to understand in real time the operator’s psycho-physical state, matching it with the situation in the ATCO is operating. The envisaged adaptive automation uses neuromeric indicators to derive information on the level workload, stress, situation awareness and vigilance.
 

Specific tool: Multiple decision support tool.

Reference scenario (narrative description): An Executive ATCO is controlling air traffic in an en-route sector in the Area Control Centre of a large Air Navigation Service Provide. During her shift there a significant increase of traffic. The ATCO starts experiencing time pressure. The Adaptive Automation functionality detects an increase of ATCO’s workload and stress. The attention, situation awareness and vigilance are also decreasing. Therefore, the decision and action selection activity by ATCO becomes harder, in particular to evaluate and compare different alternative options to solve potential conflicts. The adaptive automation functionality activates the multiple decision supporting tool, which provides, for each conflict (one at a time, based on relevance), 3 possible solutions. The ATCO goes through them and select the most appropriate one according to her judgement. In particular, for the conflict between DAL241 and ADR258 she selects the option requiring DAL241 to climb to FL 390. She communicates the instruction to the fight crew via R/T, and then she closes the supporting tool window. 

• In this simulation, different roles were involved, including ATCOs, pilots, tool designers, human factors experts and psycho-physical activity measurement experts.
• Data were collected from the physiological measurements, from the Multiple decision support tool; advice and criticisms from the participants of the simulation were also collected.
• The designers and the other experts made their reflections and proposed changes for the further development of the technology.

Illustrative Example for Maritime: A simulated futuristic decision support tool for VTS operations

Context: Futuristic scenario in which an adaptive automation functionality in integrated into the VTS system, capable to understand in real time the operator’s psycho-physical state, matching it with the situation of the VTS for monitoring and controlling the vessel traffics. The envisaged adaptive automation uses neuromeric indicators to derive information on the level workload, stress, situation awareness and vigilance.

Specific tool: Multiple decision support tool.

Reference scenario (narrative description): VTS is a marine traffic monitoring system to improve the safety and efficiency of vessel traffic in critical waterways. During a specific shift, it is observed a significant increase of traffic. The VTS starts experiencing time pressure. The Adaptive Automation functionality detects an increase in both, workload and stress. The attention, situation awareness and vigilance are also decreasing due to the increase in the workload. Therefore, the decision and action selection carried out by the VTS becomes harder, in particular to evaluate and compare different alternative options to solve potential conflicts between vessels approaching the berth at the same time. The adaptive automation functionality activates the multiple decision supporting tool, which provides, for each conflict (one at a time, based on relevance), possible solutions. The VTS personnel go through them and select the most appropriate solution according to their judgement. In particular, for the conflict between “Pride of Kent” and “Seafrance Manet”, they select the option requiring “Seafrance Manet” to wait until further notice before approaching berth. They communicate the instruction to the vessel crew, and then they close the supporting tool window. 

• In this simulation, different roles were involved, including VTS, crew members, tool designers, human factors experts and psycho-physical activity measurement experts.
• Data were collected from the physiological measurements, from the Multiple decision support tool; advice and criticisms from the participants of the simulation were also collected.
• The designers and the other experts made their reflections and proposed changes for the further development of the technology.

Key Concepts

For participants, the Focus Group session should feel free flowing and relatively unstructured, but in reality, the moderator must follow a pre-planned script of specific issues and set goals for the type of information to be gathered. During the group session, the moderator has the difficult job of keeping the discussion on track without inhibiting the flow of ideas and comments. The moderator must also ensure that all group members contribute to the discussion and must avoid letting one participant's opinions dominate. The following is a list of essential tips/guidelines to conduct a proper FG:

• The moderator should begin by explaining the purpose of the exercise and what is expected by the group, avoiding generating biases about the content of sessions.
• She/he should also address the question of how any data collected or personal data will be used and how it will not be used.
• The moderator should try to manage group dynamics and to establish a permissive environment in which:
  • hierarchical relationships are not influential in the discussion;
  • everyone feels free to contribute;
  • everyone sits equally distributed around a table;
  • no one feels judged to express thoughts and comments.

• If a participant begins to dominate, the moderator should gently encourage others to get involved and rein in the dominant participant(s).
• The moderator should not be tasked with note taking. Ideally one or two observers will do this – they should be introduced to the group and their roles explained as part of the introduction.
• Badges or panel with names of each participants should be used to facilitate both the moderator and the different attendants to refer to the others during the discussion.
• If video or audio recording is to be used – this should be explained in the introduction too.
• Refreshments should be made available and if the Focus Group sessions are lengthy – regular comfort breaks should be given.
• The moderator’s job is to facilitate, but not participate in, the discussion.
• The moderator should sum up important points at convenient moments and ensure that the majority of participants have understood them.
• The moderator (with the observers) should summarize the key themes at the end of each session, check for understanding and ask any questions that the observers feel would be useful.
• The moderator must be careful in considering that the data coming out are not totally objective, as they are the result of a partial perspective of those who have taken part in the FG. Data must be compared with those coming out from other techniques (see for example Factsheet A.15).

Benefits

Fast and cost effective method: because several subjects can be “interviewed” at the same time, FG are a fast and cost effective means to achieve relevant information as well as to obtaining attitudes, feelings, and beliefs.

Provides broad content: FG allow for an open format and they are flexible enough to handle a wide range of topics. They allow taking into account different perspectives on the solution you are imagining or evaluating.

Provides in-depth content: FG allow in-depth exploration of the reasons why the participants think the way they do and often provide insights that can be difficult, time consuming, or expensive to capture using other methods.

Builds new content through interaction: FG provide participants with the opportunity to react to, reflect on, and build on the experiences of others. This can generate new ideas that might have not been uncovered in individual interviews. It also provides the scholar with the opportunity to clarify responses, ask follow-up questions, and receive contingent answers to questions.

How It Works

Focus Groups can be developed following a process like the one described below:

• The Choose the facilitator and note taker;
• Choose and invite participants:

- If you start from an already mature solution, allows the identification of all involved stakeholders, find a representative for each of them; 
- If, on the contrary, the solution you are working on is still undefined, identify the people you consider the most expert on the topic, if possible, with different backgrounds.

• Identify the appropriate place and room-layout;
• Define the goals (Explain the goals of the FG so that all are aligned);
• Prepare the input to be given to the participants, such as:

- Guiding questions;
- Scenarios;
- Model of system or process under analysis.

• Collect the data (see one of the tips – the note taker should be different from the facilitator);
• Analyse the data.

Focus Groups are best analysed immediately after they finish. It is when things are freshest in the minds of the moderator and the observers. Other participants may be brought in to the analysis and videos/audios reviewed during that process. If possible, a transcript of the audio may be useful to prepare for later analysis at the same time. A simple report should be made of key findings after an individual FG.

Depending on what is expected from the FG, the output could be (Consider the three possible uses of the Focus Group highlighted in the Background section):

a) a set of refined scenarios, if the design process is still in the initial creative phase;
b) a list of user requirements;
c) a list of Human Performance Issues or Hazards potentially associated to the introduction of the new solution, with a proposal of how to mitigate each of them.

Illustrative Example

Aviation domain
Preventing STCA nuisance alerts in military operations

The following example concerns a Focus Group session organized in the context of a project promoted by EUROCONTROL aimed to improve the performance of a Safety Net on the Controller Working Position (CWP) of an Air Navigation Service Provider (ANSP). The Short Term Conflict Alert (STCA) is a ground-based Safety Net intended to assist the air traffic controller in preventing a too close proximity between aircraft by generating, in a timely manner, an alert of a potential or actual infringement of Separation Minima (i.e. the minimum distance allowed among aircraft flying according to the international regulations). 

When an STCA alert is activated, the radar tracks of the corresponding aircraft on the controller’s display become red. In this way, the controller attention is triggered to that specific situation that requires an immediate action to solve the conflict, i.e. an instruction to flight crews of one or more of the involved aircraft to immediately change their trajectories.

• Specifically in this project, the Safety Net was installed in a military control centre, but was configured in a way that fitted well only for civil commercial flights and did not take into account the specificities of military operations.
• Different operational experts were invited to the FG session:
• 2 Military controllers;
• 1 Military pilot;
• 2 Technical experts (with experience in the design of multi-radar tracking systems and safety nets);
• 1 HF and Safety expert acted as facilitator;
• 1 Safety expert acted as Note taker.

Several operative scenarios were analysed during the session. One of them, presented in this example, was the Formation Flight, a typical disciplined flight of two or more aircraft under the command of a flight leader. Military pilots use formations for mutual defence and concentration of firepower.

One of the problems that emerged specifically in relation to this scenario was that the STCA triggered alerts even when it was not operationally justified, because flying in close proximity was exactly what the aircraft were supposed to do. This generated many nuisance alerts that represented a big disturbance for the ATCOs and generated the so-called ‘cry-wolf syndrome’. As also shown in the diagram, during the flight in formation the STCA alerted whenever at least two of the aircraft went too close to each other.

A negative consequence then was that ATCOs asked to switch off the STCA in this and other operational situations, thus losing completely the protection given by the safety net.

During the FG, a mitigation for this hazard was identified, which consisted in turning off the aircraft transponder of all military aircraft except the flight leader one, as soon as the formation flight was going to start and only until its end. In this way, during military operations implying a formation flight, a possible proximity of the leading aircraft with other aircraft not involved in the exercise (including civil aircraft in neighbouring airspace sectors) were still brought to the controller attention by an STCA alert.   

The mix of perspectives and competences offered by the FG played a critical role in helping the project team to both identify the hazards and the mitigations associated to this important scenario, as well as a number of other hazards and mitigations.

Illustrative example – Maritime domain
ECDIS – Position Verification

The following example concerns a Focus Group session organized in the context of a project that aimed to improve the performance of the Electronic Chart Display Information System (ECDIS) on the matter of position verification.

ECDIS is a computer-based navigation system that complies with IMO regulations and can be used as an alternative to paper navigation charts. It is mandatory for all SOLAS vessels (larger than 500 gross tons engaged in international waters) to install and operate ECDIS. One of the systems feeding data to ECDIS is the Global Positioning System (GPS), which provides ECDIS with the actual position (coordinates) of the vessel at all times. For safety reasons a second GPS is usually connected to ECDIS and selected either as an alternative system in case of failure, or alongside the primary GPS.

An Officer of the Watch (OOW) is obliged to know the ship’s position at all times while the ship is at sea and in this context ECDIS has been proven to be a valuable tool. ECDIS was built to assist the user in developing various skills, among which the most important may be Situational Awareness. When the vessel is navigating in shallow or otherwise confined waters the OOW’s job is even more difficult considering the ship needs to be handled with precision and swiftly to avoid dangerous situations.

The problem deriving from the use of ECDIS is that unless the position of the ship is verified by alternative means, such as position lines (bearings, distances, contours, transits etc.) or celestial lines, the OOW cannot safely determine the accuracy of the position of the ship. Because the above-mentioned methods take time and manpower, a useful feature that could be integrated in ECDIS is the ability to cross-check the position of the ship.

To display the position of the ship on the charts accurately, ECDIS may use both GPS receivers to plot the vessel’s position. Because of the reduced accuracy of the GPS, a difference is observed between the two fixes. That distance (usually a few meters) determines the accuracy of the plotted position.

Method implementation: The FG worked towards the development of a feature that calculates the above-mentioned distance. The different operational experts that were invited to the FG session were:

• 2 Masters;
• 1 Bridge Simulator Instructor;
• 1 Designated Person at Shore (DPA);
• 1 Navigational Auditor;
• 2 Navigating Officers (OOWs);
• 1 Technical expert (with experience in the design of ECDIS systems and safety parameters);
• 1 Human Factor (HF) and Safety expert acted as of facilitator;
• 1 Safety expert acted as Note taker.

The FG conclude that the difference between POSN1 and POSN2 may be displayed and, if the user chooses, may be indicated by a user-set alarm. For example, when the Difference between POSN1 and POSN2 is 10m the alarm would sound to notify the user of the reduced accuracy of their position. The OOW can then use this information to avoid any hazards that are closer than 10m. Therefore, the OOW’s Situational Awareness would be enhanced and safer navigational decisions would be made. Errore. L'origine riferimento non è stata trovata. shows how the added feature, showing the distance between the position fixes from the two GPSs, may be integrated into an existing ECDIS system.

Key Concepts

• Goal what needs to be achieved;
• Task the individual part of an overall activities necessary to achieve a specified goal;
• Objects the agents or things used by the agents;
• Subtask the lowest level of the activity, not requiring further decomposition; what agents do, alone or with their objects, and other agents.

Thanks to the HTA it is possible to compare different approaches to support a human activity in interaction with a tool, a procedure or with another human operator. The HTA is an abstract representation of the main atomic elements of the activity: it may of course vary depending on the analyst who prepares it. For example one analyst may choose a lower level of granularity compared to another. As well as different criteria might be chosen to divide one task from the other. However having an explicit representation of the activity is as such a very important support for the design team. Different members of the team will have the possibility to use it as a reference when deciding how to develop a given design solution, be it started from scratch or redesigned. When used in a discussion within a design team, the HTA can of course be modified or refined as soon as new design issues or implementation options are identified. 

Benefits

The HTA can be used with different purposes.

• As a way to anticipate the tasks in which the users will experience more difficulties in achieving their goals or possibly end up in errors that completely prevent them from reaching such goals.
• To help spotting the difference between the so-called “work as imagined” (what is prescribed by the procedure or manual) and “work as done” (what is observed in the work setting, during real operations).
• As a basis for subsequent analyses, such as those that can be conducted with other more specialized types of task analysis (see the factsheet T08 for Tabular Task Analysis, T10 for Operational Sequence Diagram, or T11 for Critical Incident Technique, etc.) or the analysis of errors that can occur at an individual levels (e.g. the TRACEr technique explained in the factsheet T02). 
• To support function allocation, i.e. to guide the decisions about the tasks to be performed by different types of operators or by different types of hardware/software systems or by a combination of the two.
• To determine at which level of automation each task could be supported (see as an example the figure below).

On the side of limitations, it should be considered that the HTA does not capture the contextual elements affecting the performance of an operator and generally does not show in detail the equipment, tool or HMI the operator is interacting with. This is way the HTA should normally be complemented with other types of task analyses as well as with different HF techniques.

How It Works

Developing a HTA normally entails the following steps.

1. Collecting information from either the users of a tool/procedure (descriptive approach) or from the designer of them (normative approach) to know which are the tasks composing the concerned activity and how they can be grouped and organized in a hierarchy. The strategies for collecting such information normally includes: (i) reading manuals and procedures describing the activity -if available-, (ii) conducting individual interviews with the users, the designers or a combination of them, (iii) observing some of the  tasks when executed by the users of the concerned tool or procedure. 
    
2. Preparing the diagram with the representation of the overall goals, tasks and sub-tasks
    
3. Revising the diagram with representatives of the people who participated in the interviews or are either experts or developers of the manual/procedures.

Illustrative Example

Illustrative example: aircraft emergency
In the following example the HTA is used to identify the likely sources of actual or potential failure an air traffic controller (ATCO) may experience when managing an engine failure emergency occurring to an aircraft.

In order to conduct the analysis, the following steps are proposed:

Step 1: Purpose of the analysis – identify Human Performance issues, such as ATCO’s errors while dealing with a PAN-PAN situation. 

Step 2: Definitions of Goals – decide which are the outcomes of the analysis, e.g.

• design of new, automated tools to support the activity;
• writing new procedures;
• facilitating controller’s tasks and eliminating ambiguities.

Step 3: Data acquisition – done by consultation of manuals and procedures, direct observation of the work interviews with experts, performance records, or even simulations with monitoring of ATCO’s activity with system logs or flight recorders.

Step 4: Decomposition diagram (Decomposition = the process of breaking goals down into constituent tasks. It can be continued until the level of detail required is reached)Step 4: Decomposition diagram (Decomposition = the process of breaking goals down into constituent tasks. It can be continued until the level of detail required is reached).

Step 5: Recheck validity – thinking about the task, the personnel may realize that:

• events do not always occur in a standard way; 
• something which has never been known to happen could actually happen. 

An iterative process is recommended wherever possible by cross-checking between sources and professional operators, e.g. by comparing training manuals with experienced ATCOs’ judgement to obtain the most accurate and complete description of the task.

Step 6: Identify Significant Operations and Redesign the diagram if necessary.– Decide which of the tasks of the ATCO need further details and if there is any unnecessary information. If tThe ATCO’s tasks are correctly described there is no need for further decomposition or reduction of the diagram.

Step 7: Generate and Test Hypothesis concerning Task Performance – the analysis should emphasize likely sources of actual or potential failure to meet overall task goals, such as:. 

• Missed coordination between the ATCOs
• Lack of training in emergency situations
• Inadequate time efficiency 

The analyst should propose practical solutions (regarded as hypotheses) that must be tested. This is the only way to guarantee that the analysis is valid.

Illustrative example: Ship collision emergency during Strait passage 
In the following example the HTA is used to identify the likely sources of actual or potential failure an Office of the Watch (OOW) may experience when managing a risk of collision between two ships in narrow waters.

Application of HTA to ship collision emergency: 

Step 1: Purpose of the analysis – identify Human Performance issue, e.g.Officer on Watch (OOW)’s error during overtaking of the other ship in TSS (Traffic Separation Scheme)  

Step 2: Definitions of Task Goals – decide which are the outcomes of the analysis

• design of new, automated tools;
• recommending appropriate manoeuvring;
• facilitating officer’s tasks and eliminating ambiguities.

Step 3: Data acquisition – Observation of the passage, investigating records or even simulators. Monitor the ship navigation activities (ECDIS, VDR, ARPA, GMDSS communication, etc.) and check logbooks, ISM forms, Master standing order and VTS recorders (if possible).

Step 4: Decomposition diagram (Decomposition = the process of breaking tasks down into constituent sub-tasks. It can be continued until the level of detail required is reached).

Step 5: Recheck validity – thinking about the task, the personnel may realize that:

• events do not always occur in a standard way; 
• something which has never been known to happen could actually happen. 

Compare simulator trainings with experienced Officer/Master to obtain the most accurate and complete description of the task.

Step 6: Identify Significant Operations – decide which of the tasks and sub-tasks of the OOW need further details. Redesign the diagram if necessary. In this casethe task to be performed to manage a ship collision emergency during Strait passage.

Step 7: Generate and Test Hypothesis concerning Task Performance – the analysis should emphasize likely sources of actual or potential failure to meet overall task goals. 

• Inadequate Manning on Bridge
• Inadequate Policy, Standards and Application
• Lack of Communication between ships
• Poor risk assessment 
• Operational blindness
• Mental fatigue
• Inadequate experience 

Key Concepts

Tabular Task Analysis (TTA) usually follows on from HTA and is primarily used for assessing the adequacy of operator interfaces (displays and controls). 

There are typically a number of columns such as task step, indication, action or decision, feedback to the operator following the action, and potential problems or errors. The two critical focuses of the approach are the indications that tell an operator when to act, and the feedback (or lack of it) that tells the operator that the action was successful (or not).

Often when analysing a task, it is found that certain steps are not clearly initiated via unambiguous indications, or else there is indirect or even potentially misleading feedback, or no feedback at all telling the operator if the action was successful. This can lead to errors, and the TTA can be continued to look at the errors likely due to interface problems, consequences of those errors, and the necessary changes to resolve the inadequacies and prevent the errors (see Human Error Analysis). 

Benefits

TTA is a powerful analysis method, and leads to resilient design that is more robust against human error. It usually develops fixable solutions to the problems it identifies as it progresses. 

TTA allows designers, operators and Human Factors analysts to represent the task and its interfaces in a commonly-perceived format, so that agreement on required changes can be achieved. 

TTA can be resource-intensive if carried out in the early detailed stage of design (when it is most useful), since often there are not experts available to determine how the interface should be used at the required level for analysis. Nevertheless, if highly usable interfaces are desired (e.g. for safety critical operations) then the resources are justified. Once an interface is in use, TTA resources are less, but the cost of changes to the interface are correspondingly higher.

How It Works

Tabular Task Analysis
Tabular Task Analysis usually follows on from an HTA, and is columnar in format. It takes a particular task step or operation and considers specific aspects such as: who is doing the operation; what displays are being used; what feedback is given; what errors would occur, etc. The column titles will depend on the purpose of the task analysis.

The HTA/TTA combination is a very powerful one for detailed evaluation of interfaces, as the HTA gives the analyst a firm basis of understanding of the system, and the TTA can be used to systematically interrogate the Human Factors and usability aspects of the system, and evaluate problems identified on the grounds of the likely consequences or errors, initiating design alterations as needed. There are several features of particular interest to the designer / HF practitioner (and risk analyst) in TTA: 

1. the nature of the cue telling the operator to proceed to the next task

2. the feedback telling the operator the action was correct

• the speed of the feedback
• the directness of the feedback 
• the ‘confusability’ of the feedback with other signals
• the diversity of feedback signals in the interface 

3. the availability of error feedback
4. the error tolerance of the system
5. the ease of error recovery

The above may be posed as questions, which themselves are the beginnings of the error identification process:

1. What does the operator expect to see at each step?
2. Could the operator’s mindset lead them to misinterpret, question or even disbelieve certain signals? [this is especially important in the case of unusual, rare, abnormal or emergency events]
3. How does the operator know when to act? (Could the operator start too early/late?)
4. How does the operator know the act was carried out correctly?
5. Is the feedback quick enough to enable error detection?
6. Is the feedback indirect, thus potentially misleading or open to misunderstanding if incorrect?
7. Could the operator confuse signals?
8. Are there diverse signals or is the operator reliant on only one information source, and if so, what is the reliability of that information source?
9. What tells the operator (s)he has made an error? (Is error detection likely?)
10. If an operator makes an error is there an immediate or inevitable undesirable consequence? (What are the immediate and delayed consequences of each error, alone and in conjunction with other human and non-human failures?)
11. How easily is error recovery achieved? (Who recovers, and what additional errors and consequences are possible through these recovery actions?

Link Analysis

Link analysis identifies the relationships or links between an operator and parts of the system (usually controls and displays, but this can include exterior views as in ship watchkeeping tasks or pilots looking out the windshield towards an airport). The links recorded are usually physical and observable occurrences, i.e. the movements, actions or points of gaze during operations on a ship’s bridge or in another ship area, or in a cockpit or air traffic controller workstation. It can also be more ‘micro’, e.g. recording the order in which different screens or digitised items (icons, etc.) are selected when using a computer, the sequence of ‘pages’ a pilot must access from the electronic flight bag when locating an alternative airport. The links can therefore be at different levels of resolution. Overt movements are useful when reviewing Bridge layout, more micro level analysis is appropriate when evaluating the layout of the computer screen pages in the software system, assessing ease of use (‘navigability’), or the design of individual VDU pages. The type of links are first established (between the operator and the different parts of the system under investigation), levels of resolution appropriate to the investigation purposes are defined. The links observed can be annotated as in the figures below, and design choices are then informed by criteria such as the following:

Importance – the more important information should be in a central area (e.g. for an operating room / bridge / cockpit / workstation design) and in a central field of view (e.g. for critical information in a cockpit – within a fifteen-degree field of view for the pilots (or via a head-up display), usually an area common to both pilots. For a controller it should occur on the track data blocks affected. 

Function – information related to a single function should be generally grouped together

Frequency – the more frequent something is used, the more central it should be

Sequence-of-use – information should be displayed in the sequence it will be used

Usability & Navigability – information on screens should be displayed to enhance usability and navigability so that a user does not get confused or lost when carrying out normal or abnormal operations. 

Example Link Analysis
The example is from the oil and gas industry.

Example Tabular Task Analysis
The example is based on an offshore drilling emergency blowdown analysis which led to a re-design of an existing emergency panel. In this example, the platform is in a severe emergency, of the type which could happen once every thirty years (it has occurred in reality). The control room operators are supposed to initiate sequential blowdown, venting potentially explosive hydrocarbons to the flare stack from two areas – separators and compressors, but not at the same time (which could cause an explosion in the flare stack). Several issues were identified concerning the colour coding, the display-and-control panel itself (including lack of clear feedback that blowdown had been initiated), and having to remember to wait 4 minutes (no timing device or time-delayed interlock was provided). A link analysis also showed that this panel was in a secondary location away from emergency comms devices and the centralised alarm system, which is where the control room operators would need to be during an emergency, so that monitoring blowdown operations would be treated as a secondary task. 

Illustrative Example

Aviation Domain
SCENARIO: In a study of cockpit emergencies with a total of 30 real (airline) pilots, video recordings, eye tracking and detailed debriefs (walk-throughs / talk-throughs while watching recordings of their performance in a full-motion cockpit simulator), were used to examine the adequacy and sequence of use of certain controls and displays (including looking outside the cockpit). This created a link analysis and an effective tabular task analysis for several safety critical functions.

In particular, it was found during the low fuel scenario that many of the pilots only became aware of the low fuel situation quite late in the scenario, and in at least one simulation run it would have been too late to reach an airport. This was despite much cross-checking between the captain and first officer, albeit amongst certain other distracting factors and secondary tasks. The low fuel was not particularly noticeable. Additionally, when faced with a suddenly closed airport and the need to re-route to another airport, calculating the available range given the fuel remaining was a significant task. A new design was proposed with a more centralised and salient warning, including a display feature showing ‘range rings’ given the available fuel so the pilots could quickly see which airports they could reach.

Diagnosing and mitigating the effects of an electronic bus-bar failure on the aircraft during the simulated flight was also examined, using the standard cockpit displays and information available. This was found to be quite difficult to follow by the pilots, since it was effectively engineering information more suited to engineers than pilots. However, detailed discussions with the pilots led to the development of new displays for future aircraft which are more ‘pilot-centred.’ 

Identifying a suitable alternative airport in the emergency required use of the electronic flight bag (EFB), basically a computer tablet. Whereas significant design effort and learning from events goes into cockpit design, EFBs have far less aviation design input. To identify an appropriate airport could require reviewing up to 11 pages on the EFB, during which time the First Officer is not observing other indications nor the external view (out the cockpit window). Essentially, EFBs are considered as providing secondary ‘administrative’ information, yet in this case its use was safety-critical, and its navigability/usability was poor.

Maritime Domain
SCENARIO: Tabular Task Analysis (TTA) is applied to a real shipboard case study “fire on ship bridge”. In the case study, the bridge team management including master, OOW and look-out man (if any) to initiate electronic navigational equipment such as ECDIS, AIS, ARPA, GMDSS, etc. Due to the overheating of the equipment, electric panel caused fire. Various issues were identified concerning the colour coding and instructions, the display and control panels of equipment (lack of coordination, lack of feedback and time limitation that had been initiated fire on electric panel at ship bridge). In the view of above, TTA can be applied for fire on ship bridge.

Key Concepts

The key concept is eliciting expertise, whether by watching operators perform their tasks (observation), asking them to go through the tasks step by step to explain them in more detail (walk-through / talk-through), and talking as they go through the tasks explaining what they are thinking about (verbal protocols – used for more cognitive tasks).

Benefits

The principal advantage of carrying out data gathering is the development of a richer basis for design. 

What is being designed or re-designed will be more likely to be used within design and operational parameters, and will be fit-for-purpose. 
The gap between design intent and work as done will be smaller or even non-existent. This means the designed system will be more effective and longer-lasting.

How It Works

1. Observation
Observation as a technique is useful when the task aspect of interest is relatively explicit and hence observable, e.g. a manual task or interaction with a conventional control/display (hardwired) interface. Unobtrusive observation is ideal so that natural behaviour will occur, but can be difficult to achieve. 

Observation can occur with varying levels of obtrusiveness ranging from indirect (e.g. via CCTV) to direct (observer present) to participant observation (observer actually joins team and does the task).  An observer sitting there with a note book etc. can have negative effects on performance, and what will tend to be seen by the observer is ‘angel behaviour’, i.e. people doing things ‘by the book’. However, if observation is maintained for several shifts this usually fades and actual performance will be observed. Observing over the night shift can also be effective as workers appreciate the effort of someone joining a night shift, and they are often a little bored, so may be inclined to talk more about their job. 

What is imperative if an observer is present is that they explain why they are there, and that the observations are anonymous (it is often useful to show operators what is being recorded so they can see for themselves). If CCTV is used, use a system with good audio-recordings also. Observation is best used as an adjunct to other techniques. It can also feed into other task analysis techniques such as Link Analysis, Tabular Task Analysis and Operational Sequence Diagrams. If carrying out observation as a precursor to carrying out an HTA, it is preferable to observe the tasks in the field situation, where this is possible, or at least in a real-time simulation of the task.  There will however be cases when this is not possible (e.g. if the tasks are designed but the equipment not yet built, and if no simulator exists).

2. Walk-through/Talk-through (WT/TT)
A walk-through/talk-through approach utilises experts who commentate their way through their normal job or hypothetical scenarios (e.g. emergency drills), either in their workplace (walk-through: pointing to controls etc. which would be used), or off-line in an interview room (talk-through).  Either the operational expert will run through a typical procedure, or the analyst will suggest a scenario and ask the expert to commentate how they would deal with it.  The WT/TT approach can be particularly useful for examining actions in problem-solving situations, and understanding problem-handling strategies. The usual approach would be first to conduct a talk-through of a situation, and then carry out a walk-through in the actual workplace, or else in a mock-up of the workplace. There are three basic purposes of walk-throughs:

The first is the investigation of a task which is not fully understood by the analyst, because e.g. it involves the simultaneous usage of several instruments (e.g. docking a large ship in a busy port in difficult weather conditions, or for example using cockpit controls [including the electronic flight bag] and communications with ATC to locate a suitable alternative airport during a low fuel situation).  Such tasks will be far more easily understood if the analyst can actually observe a walk-through. 

The second purpose is for task timing determination, i.e. determining how long tasks are likely to take.  In such walk-throughs, the operator walks through the task in real time, as far as is possible, based on his/her knowledge of system response times, etc. The validity of these times for the real situation is something that must be judged by the analyst, and is difficult to give guidance on.  However, it may be prudent to add at least 20% to timings carried out during a walk-through, when estimating timings for response during an incident. This will allow in part for factors such as unpreparedness, etc.

The third purpose of walk-through is for evaluation of the interface. In such an evaluation, the operator will walk through the actions fairly slowly (e.g. on a ship bridge, or in its engine room, or in a cockpit simulator or ATC Operations Room), and the analyst may need to ask questions, for example why certain interface elements are used and others are not used. The results of a walk-through are usually used to help develop Hierarchical Task Analysis (HTA), Tabular Task Analysis or Operational Sequence Diagrams. 

3. Verbal Protocol Analysis 

Verbal protocols are an oral commentary on what is being done by the operator whilst (s)he carries out the task.  Such verbalisations are useful particularly if observing highly skilled activities in which the operator is also using knowledge and carrying out mental operations which would otherwise be unobservable.  

Retrospective protocols, i.e. made after the task is finished, are not recommended. Verbal Protocols will not be useful if the operator cannot verbalise how (s)he is carrying out the task, which may be the case with very expert tasks. Verification of protocols as being valid verbalisations of how the operator performs is achieved by carrying out the exercise with multiple operators, though in some cases there may be variation (e.g. air traffic controllers each have their preferred solutions to certain problem situations). However, this method is a useful supplement to observation particularly if a lot of unobservable mental processing is taking place. The analyst should, however, beware of verbalisation interfering with the task performance. 

This method entails the operator ‘talking through’ what they are doing and what they are thinking, what information they are using, what criteria they are using and any uncertainties they may have when making decisions. As such it may be seen as a far more detailed version of the talk-through technique. The protocols must be concurrent (i.e. spoken at the time of carrying out the task) rather than retrospective (remembering what was being thought during the task, after the task is finished), and this requires a little training. Not all operators can ‘protocol’ satisfactorily. The analyst should not interfere by asking questions of clarification during the protocolling itself, as this can change the thinking pattern. To help this, the analyst should already have a reasonable knowledge of the system and the meaning of all the protocols elicited. Debriefs can occur afterwards to help the analyst understand whatever they did not follow. 

A good ‘warm-up’ exercise for operators is to ask them to go verbally protocol a simple task such as making tea or carrying out an operation on their smart-phone, to get them into the feel of verbalising. The verbal protocols obtained must then be represented and transcribed into a more standardised format, prior to analysis. Verbal Protocolling is useful as it is direct and one of the only means of eliciting such expert information. However, sometimes expertise is simply not accessible in language terms, i.e. if the operators are using mental imagery or mathematical functions, or if they really cannot say how they do the task because it is too intuitive a process. 

Illustrative Example

Aviation domain
SCENARIO: Some years ago, prior to the 9/11 attack, an ATC simulation was held to determine how quickly and safely an air traffic sector of airspace could be cleared of all aircraft if required, due to either terrorist threat (including cyber-attack), adverse weather (Volcanic Ash or major thunderstorm), or infrastructure failure (total power loss to ATC systems or multiple radar failure etc.).

After reviewing the procedures, and a talk-through with one set of air traffic controllers (ATCOs), a simulation was run with a seasoned set of ATCOs wherein they did not know the nature of the exercise. After ten minutes of normal simulation, the screens suddenly blanked. At the time the controllers had paper strips available as a back-up to what was on the radar screens, and so after verifying that it was a complete instrumentation failure (and not a simulation glitch) they began contacting all the aircraft in priority order according to closeness and vectors between the various aircraft. Because the ATCOs had to work together, there was a high degree of communication and checking between them, and so the criteria they were using (e.g. intended airport destination, on board fuel destination, capacity of surrounding vectors etc.), were clear to the observers. The simulation was effectively a live walk-through and observation with a high degree of verbal protocolling. The outputs were a refinement of the procedures and training, as well as design recommendations relating to a clear signal that it was instrumentation failure and the screens were not coming back on (during an actual loss of power event, ATCOs delayed action for several minutes believing the system was about to be restored), and back-up system information requirements for the ATCOs in case of an actual event. Initially the ATCOs believed they had covered all the aircraft, but in fact they had missed one. This changed their minds on the need for emergency continuation training and more information back-up.
 

Maritime domain

SCENARIO: Walk-through/Talk-through (WT/TT) technique is applied to real-time simulator based ship passage through the narrow canal. The full-mission bridge simulator was used to determine how quickly and safely Master of ship and officers on watch (OOW) could be clear of the ship if required, due to either restricted visibility (dense fog or rain), adverse weather or fundamental failure (total power loss, ECDIS failure, ARPA radar failure, etc.).  

The simulation procedures were explained to Master and OOW along with talk-through, a real-time bridge simulation scenario covering ship passage through the narrow canal was run. Master of ship and OOW did not know the nature and content of the exercise. Normal simulation was run about five minutes and then the screen of ECDIS and ARPA radar were suddenly switched off. At that time, the weather condition was changed to restricted visibility (dense fog – visibility was arranged less than 1 nautical mile). The master and OOW were not aware of the situation but it was fundamental failure associated with ECDIS and ARPA radar. Even though the screens were run up, they could not plot the other ships (in order to predict risk of collision) due to failure of ECDIS and ARPA radar.  They began contacting all the vessels in priority order according to closeness and vectors between the various vessels. The bridge team (Master and OOW- helmsman and look out) had to work together since there was a clear communication and monitoring between them. The criteria such as closest point to approach (CPA) other ships and time to closest point to approach (TCPA) other ships were quite importance to avoid collision in restricted visibility. The real-time bridge simulation was effectively a live walk-through and observation with a high degree of verbal protocolling. The findings of the application were a refinement bridge procedure, conduct of vessels in restricted visibility and improvement of training quality on-board ship. Regular maintenance of electronic navigational equipment (ECDIS and ARPA radar) was another recovery action to avoid fundamental failures (during failure of ECDIS and ARPA, Master and OOW on the bridge delayed action for a couple minutes considering that there was a power supply problem). Finally, Master and OOW realised that they did not have enough experience in case of emergency such as blacking out electronic navigational equipment. Therefore, they accepted the need for emergency practical training and more information back-up.   
 

Key Concepts

The key components of an Operations Sequence Diagrams (OSD), usually represented by a table with a number of columns suited to the task, are as follows:

1. Time, indicating the exact moment or the timeframe in which the task is performed.

2. The goal of the operators, if this will evolve during the scenario, for example if the situation degrades significantly as time progresses (e.g. in a ship collision scenario the goal may start as collision avoidance, but may end with evacuation)

3. The different personnel involved in the task, including their location if not co-located

4. The system status - what is happening in the system and/or environment, whether or not the operators are aware of it 

5. The key information sources they have available 

6. Decisions/Actions/Communications (who does what in each time-slot; actions can include ‘monitoring’ if nothing active is happening)

7. Secondary tasks or distractions if appropriate

8. Analyst comments

Benefits

Operations Sequence Diagrams (OSDs) bring the task to life, and make more understandable the team’s tasks and the required coordination, as well as points in the task which may be vulnerable to error or be under time pressure or stress or significant uncertainty. 

An OSD can help model the task and identify the following:

• The required task ‘rhythm’ for effective coordination between human operators / crew members
• Discrepancies between operator understanding and system state
• Key decision points
• Critical actions
• Areas where they don’t know what the system (automation) is doing and/or why
• Potential ‘pinch points’ where there is little margin for error due to time pressure
• Critical communications and communication vulnerabilities
• Critical information sources / displays as well as areas for improvement
• Staff shortages / criticalities

If the task is overly complex, and particularly if the nature of the task is recursive (so that there are ‘loops’ in the task), then this can become difficult to represent in an OSD. In some cases, e.g. where diagnosis is required or there are complex alarm systems involved, a tabular task analysis may be required to ‘drill down’ into how the interface is used by the human operators. 

How It Works

The task is described in linear time according to what each person does or needs to do, and what information sources they have available, in relation to how the task or incident unfolds. Usually this requires prior information gathering and/or working with operational experts to develop the OSD (see illustrative example). The table below illustrates the typical format of an OSD table with the key components to analyse. The content that should be included is briefly described in each column. 

Illustrative Example

Aviation Domain
Wake Vortex Case Example 

Operational Sequence Diagram (OSD) was used in the SAFEMODE Wake Vortex case study. Incidents relating to wake vortex were reviewed, and then three commercial airline pilots who had experienced such incidents were interviewed, first using the Critical Incident Technique (CIT), and then to develop an OSD (both CIT and OSD were developed during a single day with the pilots), as shown overleaf.

The ‘time’ element in this OSD is shown in qualitative terms, as wake events happen suddenly and are often over in 20-30 seconds. The human operators are the captain and first officer, and key instrumentation here relates to the status of the autopilot (A/P) which is usually flying the aircraft (a/c) in the cruise portion of the flight. ATC (Air Traffic Control) represents an outside agency who are generally directing the flight but who may be unaware of the existence of a wake threat. 

The pilots in this study found the OSD to be an excellent format for representing this type of event and their role and actions within such events. Some of the key learning points were as follows:

• Currently the pilots build an image of the surrounding airspace by using R/T information, contrail, TCAS information and wind information + geometry of a/c pairs, if they are not informed by ATC. 
• They would welcome a warning /alert from ATC so they could be prepared for such events, thus avoiding ‘startle response’, and enabling them to secure the cabin and ‘cover the controls’ in case autopilot disengages
• They do not know the criteria for A/P disengagement, which appears to be aircraft type and age dependent
• Existing procedures are useful (though there is little time to use them) as long as pilots are aware it is a wake encounter and not an instrument failure
• The flight crew plan for manual intervention (in case of wake expectation) but keep the A/P engaged unless a corrective action is required. The need for corrective action is down to judgement.
• During a sudden unexpected wake, the pilots may feel the induced roll is more significant than it actually is.
• During the event there is less looking outside and more focus on the cockpit instrumentation and Captain-FO exchanges. However, if there is a head-up display (e.g. 787) it still allows for peripheral vision/ outside view.
• The ‘pilot flying’ checks primarily the flight parameters (e.g. climb/ descend, bank angle, speed, attitude)
• The Pilot not-flying would continue to monitor the flight path.
• They also check the Flight Management System to make sure it does no go into a degraded mode.

Maritime Domain
Fire Discovery Procedure Example 

OSD can be applied to a great number of operations taking place on board ships for representing the complex interactions between crew members, passengers and ship systems. In this specific case, existing procedures used by Operators have been fused to describe generally established procedures for ship fire emergency evaluation.

Similarly to the Aviation domain, time indications are only qualitative, since they depend on a variety of factors as extent of fire, reaction time of all involved personnel, time needed to evaluate the situation, etc.

The main actors in this assessment phase are:

• First Notifier: a passenger or crew member that suspects fire onboard and activates alarm
• First Responder: a crew member with fire fighting or safety skills on duty, able to locate fire or to assess its non-existence
• Command Team: the Officers taking decisions. They can be Ship Master, OOW or other Officers.
• Assessment Team: fire fighting personnel on duty, able to assess extent of fire and severity of the situation

The procedure described in the following table can be used for many purposes, among which we mention training, revision of already existing procedures, determination of the ‘bottlenecks’ and evaluation of the uncertainties the Command Team can have in any phase, thus triggering the addition of new systems or procedure changes able to better address the risks identified.

Key Concepts

CIT is a flexible interview approach – as such it requires no prior training.

A set of questions are asked. These are best undertaken with open-ended questions, and the interviewees should be encouraged to state what happened and why in their own words (their narrative).

CIT can be single or group interview, or both.

A set of ‘factors’ may be used to explore what was affecting human performance during the event. These factors can be used to inform Human Reliability Analysis / Risk Analysis if that is to be carried out. 

The answers require no special codification, though they often feed into other task analysis methods such as Hierarchical or Tabular Task Analysis, or Operations Sequence Diagram (see aviation illustrative example), or directly into designer consideration.

Benefits

CIT can really add another dimension to a task analysis that is purely based on what people ‘should’ do. As such it adds important ‘colour’ and understanding of how stress plays a part. It can highlight what information (e.g. from displays or communications or observations) the people in the incident really paid attention to, and what they ignored, which is valuable information for a designer or procedure writer. It can also give useful insights into how actions between different people in the incident were coordinated, and the time it took for things to happen or be affected. It may identify issues or factors the designers, procedure writers and/or trainers had not previously considered. Incidents and accidents are where the design, procedures and training – in fact all the Human Factors in the system – are tested ‘through fire’, and you see what helped them make it through such challenging events.

More generally, CIT can also prevent ‘hindsight bias’, whereby people (after the event) wonder why those involved in an incident or accident didn’t simply ‘do the right thing.’ This is usually because in the heat of the moment, the ‘right thing’ was not clear. A relevant example of this was the landing of a plane on the Hudson River after a bird-strike to both engines. The flight crew were initially criticised for not trying to fly to the nearest airport. Only after further simulations, taking into account the required thinking time to make a decision, was it realised that if they had done so the plane would have crashed into a populated area with considerable loss of lives. A CIT would have identified this thinking-time component, often underestimated by designers and procedure-writers.

The disadvantage of CIT is that it is subjective. Therefore, other more objective methods may be employed (e.g. prototyping and simulation) to verify any design decisions informed by CIT. Similarly, CIT focuses on one or two events that have actually happened, and insights should not be give disproportionate weight, as there are likely to be many other potential incident /accident scenarios that have yet to happen. Nevertheless, the fact that something has actually happened means it could happen again, and so some kind of design or other restorative action will be required. CIT will help ensure that the restorative ‘fix’ addresses the human element, making it more acceptable to future operators of the system.

How It Works

The basic approach is very simple, and involves asking a question along the following lines: 

If it relates to an incident or accident that has actually happened, the opening is along the lines of the following:
Please tell us in your own words what happened.

Typical follow-up questions are as follows:

• Can you tell me what you were doing before the event? Were things quiet? Busy?
• When and how did you notice that something was happening?
• How did you react?
• How did others around you act?
• How was the situation resolved?
• How did it affect you at the time, and how did you feel about it afterwards? 
• Were you surprised or startled? 
• How unsafe did it feel, on a scale of 1 to 10? 

Once the facts have been established, it is possible to ask about the factors that may have influenced their performance. This is best done first with an open question, so that they tell you in their own words, and then they can be shown a list and asked if any of the following factors listed affected them.

• Startle/surprise
• Unfamiliarity with handling a wake event
• Workload (high or low – please specify)
• Displays
• Team coordination
• Experience
• Communications/support
• Vigilance / fatigue
• Visual circumstances
• Weather conditions 
• Time of day / night-time

The final questions shift the interview from what did happen to what could have happened:

• What could have made it worse on the day?
• What might have made it better?
• If a similar event happened again, would you do anything differently?
• What would you have liked to have been aware of: 
  • before the event 
  • during the event 
  • after the event
• Anything else you’d like to add?

Illustrative Example

Aviation Domain
CIT was used to interview three pilots who had experienced Wake Vortex events, wherein their aircraft encountered the wake from an aeroplane ahead of them, creating moderate to severe turbulence for their aircraft. Specific variants of the CIT generic questions were asked:

• Were you aware of the other aircraft? 
• Did ATC warn you?
• Could you see their condensation trail? Could you see the other aircraft?
• Did you consider you might encounter a wake or was it a sudden surprise?
• Did the autopilot disengage automatically?
• Did you take manual control? How easy was it to control the aircraft?
• Did ATC give instructions/information during or after the event?
• Did you inform ATC?

An example of the type of material gained from one of the interviews is as follows:

The event occurred during daytime, and during a period of low alertness.

The captain saw the contrail of the aircraft ahead “coming down” which was a “surprise”.

The captain told the First Officer (FO) to put the seat belt sign on and move the seat forward to get ready for the encounter. No cabin crew warning was given.

The rolls of the vortex were visible: “rotating tubes” in clear air.

On a scale from 1 to 10 (10 being the most unsafe), the captain categorized the event as a 7 or an 8, mentioning “significant stress”. 

The roll was “sensed” as a 15°-degree angle of bank, although it was 8°.

The startle of the encounter had passengers screaming but no injuries were recorded.

After the event, the flight crew contacted ATC. 

Currently the pilots build an image of the surrounding airspace by using R/T information, contrail, TCAS information and wind information + geometry of aircraft pairs, if they are not informed by ATC. 

By having prior ATC instructions/warnings, e.g. a verbal alert several minutes before the possible encounter, pilots believe they would overcome or better prepare for encounter as it may occur during “low alertness” states.

Training should expose flight crew to en-route wake encounters both theoretically and practically. The pilots think the training will most probably focus on recovery as it is hard to simulate the “startle” effect. 

The top 6 factors (no priority order) identified by the group of pilots were as follows:

1. Seatbelt sign on / cabin crew info (would minimize the outcome of the wake in terms of passenger and cabin crew disturbance /potential injuries)
2. Covering controls (being ready to take control)
3. Vigilance / Situational Awareness (SA) / circadian rhythm and arousal (monitoring throughout the flight depending on flight crew alertness)  
4. Startle effect (slower reactions or over-corrective input)
5. Robust Autopilot (depending on aircraft type it might disengage faster)
6. IMC (in cloud) / night (less “outside” view / lower alertness)

The CIT in this case was part of a sequence of task analyses for the Wake Vortex study. You can see how it informed the Operational Sequence Diagram (OSD) by reviewing the case study in the Factsheet for OSDs.

Maritime Domain
While approaching a harbour, a bulk carrier was met by an escort tug who was to assist the vessel to a mooring station. When approaching the harbour entrance, the vessel experienced a sudden loss of steering. A loss of steering is an unexpected event that can be manifested by several mechanical issues (e.g. blackout, engine gear box failure, steering engine failure on rudder, rudder jam, loss of starting air pressure (due to many consecutive manoeuvres). 

Specific variants of the CIT generic questions were asked:

• Were you aware of any ongoing repair or maintenance orders that might affect steering control?
• Were you in communication with the engine control room (ECR)?
• Were you aware of any bridge alarms that would indicate abnormal conditions?
• Were you confident about the manoeuvring safety zones and traffic conflicts nearby?
• Did you communicate the issue at first indication to others on the bridge or in the ECR?

An example of the type of materials gained from crew interviews identified reasons for steering failure and how human factors, automation and work practices may provide procedural, pedagogical and/or mediating actions. 

Black out:

The emergency generation should start and provide enough power for essential equipment like the steering engine. If the power management system is working and there are auxiliary engines ready and in automated mode, they should start, connect and equipment should by sequence automatically start. The emergency generator stops automatically when auxiliary engines are up and running and can take the load.

Routines and check lists should be in place to get everything up and running. Communication with the bridge is an immediate action to inform about the situation.

If it is a critical situation, narrow passage, arrival or departure the ECR must be manned by the chief engineer who usually is responsible for informing the bridge about the progress.

Problems with the steering engine:

A hose could break causing a rapid drop in hydraulic pressure. Usually there are two electrohydraulic motors and in critical situations two are running. If one stops the other should automatically start. 

Bridge should be informed about this incident.

Loss of remote steering:

If the remote steering from the bridge is not working the steering engine can be operated from the steering gear room. 

The hydraulic valves are then hand operated according to the bridge instructions. A rudder indicator is fitted in the steering gear room and the engineer is following the instructions from the bridge.

The communication is through a fixed mounted emergency phone or using VHF.

Checklists and routines need to be practiced regularly (re. SOLAS)

Problems with the rudder:

Mechanical failure causes the rudder to stick.

Problems with the main engine:

Numerous reasons for the main engine to stop. 

• Fuel quality 
• Shutdown alarm on an auxiliary engine
• Could also be a mechanical failure which forces a stop of engine
• Leaks of fuel, lubricating oil, cooling water etc.
• Problems with the gear
• Mechanical failure
• Hight temp lubricating oil, high temperature lubricating oil.

Problems with basically any critical transmitter

The top factors (no priority order) identified by the bridge and ECR teams:

• An emergency plan with the escort tugs (if within arrival and departure protocols) should be reviewed (similar to a pilotage plan)
• Better communication between bridge and engine control room
• Clear understanding of operations and detection of abnormal sensor data outputs
• Training for immediate control (emergency steering system of the steering engine)

Bridge team better aware of the sensor mimics and alarms from the Engine Dept. 

Key Concepts

The real time simulation as an evaluation tool allows human operators to test and validate new concepts and new automation procedures. To be able to draw conclusions on the benefits of the proposed new concepts, procedures and communication means, human performance data have to be collected.

Those data can be:

Objective Measures: 

1. E.g. number of flights controlled in a sector of airspace in a certain time period
2. Interactions with the system (e.g. number of mouse clicks),
3. Number of instructions given by controllers to aircraft pilots
4. Number and length of verbal communications 
5. Data on the user’s gaze (using eye movement tracking)

Subjective Measures: 

1. Perceived workload (e.g. Bedford Workload scale, NASA TLX (NASA Task Load Index), ISA (Instantaneous self-assessment of workload) rating scale, etc.) 
2. Perceived situational awareness (e.g. China Lakes and SASHA Situation Awareness for [SHAPE- Solutions for Human Automation Partnerships in European ATM])
3. User’s trust (SATI-SHAPE Automation Trust Index)
4. Potential for Human Error 
5. Teamwork and Cooperation
6. Users Acceptance (CARS- Controller Acceptance Rating Scale)

This and further information on HP measures can be found on the eHP repository on the Eurocontrol website. 

Benefits

The benefit of these types of simulations is that, apart from it being a safe way to conduct experiments, certain situations can be repeated; events can be injected to simulate emergency as well as non-nominal situations. The system can simulate the future, which allows comparing the expected environment with the reference environment. This makes the quantification of the benefit possible. The data of the simulated reference scenario can be compared with the simulated future scenario in a stable experimental design allowing concluding on the direct impact of the proposed solution. 

A more general benefit can be user acceptance by the user community. For example if controllers or pilots participating in a RTS of a new system design are genuinely pleased with it, this acceptance can spread through the rest of the user community. 

There are few disadvantages of RTS and other simulation approaches, other than the fact that they can take a lot of preparation and resources. They are best run with licensed personnel (e.g. air traffic controllers and line pilots), though sometimes for more experimental or futuristic system simulations, test pilots or controllers involved with future systems design may be involved. 

With any RTS usually a battery of measures is used (objective and subjective), in an attempt to triangulate between them to determine how performance is being affected and where to improve. But an advantage with real subjects (licensed personnel) is that they can often tell at an overall level whether the system will be an improvement or not, and where it may need to be ‘tweaked’ so that it will deliver best performance. 

Usually simulations are not so useful for safety issue quantification, as even a large-scale simulation may have in total less than a hundred ‘runs’, and unless there is something radically wrong with the design, safety issues do not usually appear during the simulation campaign. Nevertheless, they can happen, or else precursor events may arise which could, if left to develop, lead to safety issues. The point here is that a simulation without any observed safety issues does not guarantee that the concept under development is indeed safe. That is for other approaches to determine (e.g. HRA and risk analysis).

How It Works

Validation simulation

Pre-simulation
Human performance objectives, success criteria (evidence) and the respective measurement have to be identified and formulated.  The simulation is planned as an experiment ideally simulating a reference scenario with the solutions scenario, respecting basic quality indicators for good experimental design. Procedures and operating methods shall be defined before the simulation. 

During-simulation
The simulation is conducted with the human operator interacting with the system applying the defined procedures and working methods, etc. Data on the prior defined measurements are collected. The collection of data can be performed by the system automatically, by observers, and by the human operator. The human operator files his/her feedback in pre-defined questionnaires and post-simulation run debriefs. 

Post-exercise
In post-exercise questionnaires (usually administered immediately after each simulation run) evidence for Workload, Situational Awareness, Trust, HMI Usability and System Error-related questions are collected. A post-simulation questionnaire can be used to gather more qualitative data and insight on the overall perception of the runs on a certain concept (e.g. trust, usability and acceptance). As compared to the post-exercise questionnaire, the post-simulation questionnaire (after the entire simulation exercise) contains more open questions, allowing the ATCOs to detail their overall experience. Observations can be carried out for each of the runs for all active working positions. The observers take notes related to any system errors, pilot errors and ATCO errors, as well as of the working methods of the ATCOs and the communication. Structured Debriefs shall be conducted after every individual exercise run, with a more detailed debrief at the end of the day. The debrief allows the human operators to express their feedback for the previous run in relation to the position they were assigned to, supported by the expert notes. The final debrief can follow a wider approach, allowing each human operator to give a general feedback on the runs completed throughout the simulation.

Post simulation 
The data have to be analysed and interpreted to be able to provide a recommendation to the decision maker on the concept implementation.

Prototype session
A prototype session might be less following a tight experimental schedule and be more agile and able to react to proposals of system improvement. The techniques to collect data can be the same as for the simulations; the topic however might focus more on HMI usability.

Illustrative Example

Aviation domain
The aviation example presented here derives from a study conducted in a projects of the SESAR Programme (Single European Sky ATM Research), dealing with the innovation of the Air Traffic Management in Europe. In the case of this specific project, the innovation consists in the testing of a set of enhanced arrival procedures intended to facilitate the reduction of the environmental impact (e.g. noise, fuel) and the increase of runway throughput.

Real-Time Simulation
One of a series of Real-Time Simulations (RTS1) assessed the application of the Increased Glide Slope (IGS) concept on the Paris Charles de Gaulle (CDG) airport and with an approach environment. The enhanced arrival procedures were simulated to be flown with GLS (GBAS –ground based augmentation system- Landing System) while the conventional approaches were simulated to be flown with ILS (Instrument Landing System). The enhanced arrival procedures were supported by an ATC support tool – the Optimised Runway Delivery (ORD). The tool was designed to reach the goal of an increase of runway throughput, by helping the ATCOs to determine and achieve the required a/c spacing/separation, using enhanced arrival procedures. The Human Machine Interface (HMI) of the ORD shows the initial and final target distance. The initial target distance indicator considers the compression effect. While the final target distance indicator shows the distance that has to be achieved at the threshold.

The aim of this exercise was to assess:

• the usability and acceptability of the IGS procedures in a large and congested airport
• the usability and acceptability of the sequencing & separation tool (ORD) that facilitates the management of mixed ILS and IGS approach procedures by not degrading workload and situational awareness and without reducing capacity
• the impact of the IGS communication exchanges/ phraseology proposed for IGS procedures
• the usability of the HMI for the IGS procedures, and
• the acceptability of the number of aircraft flying the IGS procedure.

Three traffic samples were simulated:

• A reference scenario with no enhanced arrival procedures (RECAT)
• A traffic sample with 25% of the aircraft being capable of flying IGS. (EAP 25%)
• A traffic sample with 100% of aircraft being able to fly IGS. (EAP 100%) 

The aircraft labels presented on the controller working position contained the information about the aircraft equipment: G for aircraft able to fly IGS and I for aircraft able to fly ILS only.

Objective and subjective data were recorded during the validation exercise: Objective data included the number of aircraft on frequency, the number of ATCO instructions, the number of separation infringements etc. Subjective data included workload, situational awareness and the usability of the concept and tool.

Objectives on Safety, Operational Feasibility, HMI usability, ATC support tool usability and the appropriateness of the phraseology were formulated.  

Validation Objectives
The following objectives were formulated:

1. To confirm that Increased Glide Slope (IGS) approach procedures do not negatively affect safety from ATC perspective
2. To confirm that the IGS is operationally feasible from ACT perspective
3. To confirm that ATC separation delivery support functions for IGS are usable and acceptable
4. To confirm that ATC HMI for IGS is usable and acceptable for the controller
5. To confirm that the phraseology used by ATCOs for IGS is clearly understandable

For the sake of brevity, only Objective #3 will be considered here for an illustrative purpose.

Success Criteria for Objective #3
The following success criteria were defined:

• The usability of the support tool (separation tool) is rated as being acceptable 
• The support tool (separation tool) is rated as being useful  
• The support tool (separation tool) supports the application of the IGS procedure 
• The ATCOs trust the support tool (separation tool) that facilitates the application of IGS 

Excerpt of the Results
The SHAPE Automation Trust Index (SATI) was used to provide an assessment of trust and usability of the ORD tool and was hence applied only for the runs in which the enhanced arrival procedures were instructed. The index encompasses six questions rated on a Likert scale from 0-6, corresponding to answers from “never” - “always” (negative to positive).

After breaking down the statistical data for each of the six questions of the scale, the results indicate a stable response rating for both INI and ITM that were confident in working with the ORD tool. They found it reliable, robust and easy to understand.

Excerpt of the Conclusions
The ATCO HMI had a sequencing and separation tool (ORD tool) implemented to facilitate the reduction of separation of two aircraft that are on different glide slopes. The findings show that the tool supported the concept and improved ATCO performance. The ORD tool facilitated the common situational awareness, which allows the INI to take over some tasks from the ITM to offload her/him if needed. The ORD tool and the associated procedures were considered useful and easy to use. 

Excerpt of the Recommendations
The ORD tool shall be fine-tuned to take the wake vortex figures in combination with the assigned glide slope into account. The availability of the ORD tool and its benefits should be supported by further HMI improvements.  The HMI should be consistent and indicate – as it does for ILS – the area where the aircraft shall intercept the GLS path. This would result in a reduction of the communication/coordination needs between INI and ITM but also the communication between ITM and pilot. 

Maritime Domain
Even though the structure and scope of the simulation may differ, the maritime sector also uses real-time simulations for testing purposes. An example is provided below:

Real-Time Simulation
RTS (Real-Time Simulations) were used to evaluate the risk of ship collision during passage through the Istanbul Strait. The case name is “MV Vita Spirit” which had collided with a mansion on the coast of Istanbul Strait. The ship lost control due to main engine failure during passage through the Strait. The case was simulated on the NTPRO-5000 Full Mission Bridge Simulator (RTS). The passage through the Strait procedures and environmental condition were simulated under control of the VTS (Vessel traffic services) operator. The enhanced approaching and passage procedures and conditions were supported by simulator operator (acting as VTS). In the case study, ship particulars (size and type of ship, engine power, manoeuvring ability, thrusters, anchors, rudder type and other specific features), other ships passing through the Straits, passenger ships, fishing boats and sailing boats were inserted into the scenario. The BRM (bridge resources management) team consisted of ship master, pilot, OOW (officer on watch) and helmsman. This simulation was performed in a real-time environment including time of day (early morning), current and wind, weather condition, traffic congestion and VTS control. The simulation training was performed for one day followed by four measured sessions. One operator working position was simulated. The bridge team consisted of pilot, master, OOW and Helmsman in one group, and the other group consisted of master, OOW and helmsman (without pilot). 

Objective and subjective data were recorded during the exercise. Objective data included the number of ships passing through the Strait, number of passenger ships, fishing vessels and boats, time interval of CPA (closest point to approach) and TCPA (time to closest point to approach), response time to collision avoidance, tugs positioning distance, etc. The subjective data included, situational awareness, communication, collaborative working quality on the bridge, stress, fatigue (if any), etc.

Validation Objectives
The following objectives were assessed:

• To confirm that taking a pilot enhances safety of navigation in the Strait passage.
• To confirm that VTS do not negatively affect the decisions of ship master or pilot (if any) during passage or collision avoidance manoeuvring.
• To confirm that the TSS (traffic separation scheme) is operationally feasible from perspective of ship master.
• To confirm that terminology used by VTS operator is clearly understood by ship master. 

Excerpt of the Result 
Criterion-referenced testing was applied to a particular group of pilots and master during passage through the Istanbul Strait. The scores of the individuals were ranked, and the results permitted comparison of performance amongst individuals or across a group of individuals assumed to be similar in make-up and knowledge. In view of the findings, it was observed that the ship which took a pilot during Strait passage completed her passage much safer than the ship without a pilot. There were. 

Excerpt of the Recommendations 

• The quality of collaborative working on the bridge should be improved to minimize risk. 
• Situational awareness and communication with VTS should be improved for the BRM. 
• Coordination between master and VTS operator is critical in order to avoid collision/contact during passage through the Strait. 
• Tug assistance should be available in case of emergencies.      

Key Concepts

The key concept of eye tracking is to be able to identify exactly what someone was looking at, when, and for how long. Eye tracking can be used in isolation, or in addition to other methods and techniques such as simulation or as part of a task analysis.

Benefits

• Eye tracking outputs can be particularly powerful when communicating results. This is especially true when working on multi-disciplinary projects. A diagram clearly showing how often someone looked at something or the scan path they followed in order to make a decision can paint a very clear picture to non-specialists. As already stated, eye tracking outputs can be used to give us an insight into peoples’ cognitive processes which they would otherwise be unable to communicate.

• Eye tracking outputs, specifically video outputs, can be used during interviews as an aid to the interviewee: “can you explain what you were doing then?” or “what was going through your mind during this action?” The prompt of seeing their actions again ‘through their own eyes’ can often trigger the interviewee to remember specific thoughts or feelings and allow a deeper insight into the actions at that time.

• The main downside of eye tracking is the requirement of specialist equipment, and the need acquire a specific product for analysing the data depending on the eye tracker product. This is a substantial outlay, but they are also available for hire via short-term licenses. The actual use of the eye tracking equipment also has some downsides. Depending on the product used, some of the head mounted glasses-type eye trackers can obstruct peripheral vision as the frames can be quite thick. Some newer models are frameless for this reason. Also, the equipment requires calibration before use, and can easily de-calibrate if the user touches the equipment or moves suddenly.

• Another point for consideration is the analysis of data. Eye tracking produces large quantities of data that quite often requires substantial ‘cleaning’ before they can be analysed. This takes time, but when this is done the actual analysis is fairly straightforward and most of the software packages are user friendly.

How It Works

Eye tracking requires specialist software to both gather and analyse data. The good news is that there are now some extremely good quality, affordable options, which has opened up this area of analysis, which in turn has led to more research in the area and an increased understanding of outputs and metrics.

Typically there are two types of eye tracking device: head-mounted, or screen / panel mounted. 

• Screen or panel mounted devices are typically used where the operator is in a fixed position and normally looking in one direction. These devices are used a lot for User Experience (UX) and web design applications to elicit where a user is looking in order to design webpages more effectively. These devices are also sometimes used in activities such as car driving, as the user is sat in one position and typically looking in one direction. 

• Head mounted eye trackers are worn by the operator and are a lot more versatile as the operator can move freely. These head mounted trackers can resemble a pair of glasses, and have multiple cameras tracking the operator’s pupils and eye movements, and also filming what they are looking at. Both types of device can be used in lab-based or applied settings since they are relatively non-intrusive. 

Gathering data is fairly straightforward and depending on the device, can produce many different metrics. The most common eye tracking metrics are based on fixations and / or saccades. Fixations are discrete stable points of where the eye is looking for at least 300ms (and therefore visual information is processed by the brain), and saccades are defined as the eye movements between these fixations (where visual information is not processed). Typically, the first stage of analysis would be to separate the area the operator is working in or looking at into ‘areas of interest’. These then allow you to focus your analysis. Fixation metrics can include number of fixations, fixation durations, total number of fixations in one area of interest, fixation density, and repeat fixations. Fixation metrics can tell us a lot about the operator’s engagement with certain tasks or information. For example, if an operator is focussing on an area for a long time, it could indicate a higher cognitive effort. If an operator is fixating on many different points, it could indicate that the information the operator requires is scattered around, and we should aim to work out why, and if the presentation of information could be improved.

Saccade metrics can provide us with information regarding decision-making, but typically saccades and fixations are used in combination to provide scan-path data. Scan-paths can give us insight into how people go about looking for information, and allow us to analyse how information is gathered and used in certain situations. Scan-paths can be compared between operators, and between novice and expert users to understand the differences between approaches and the impacts these may have. We may be able to identify certain scan-paths that are always straight, and quick, indicating that this task is efficient. Conversely, we may be able to identify scan-paths which are erratic which may help us to understand any aspects of the task or information provision which isn’t currently working as well as it should. These metrics can provide us with some powerful insights into how work is carried out and can greatly assist with redesign of interfaces, displays, and tasks.

Illustrative Example

Aviation Domain:
As part of a larger project, we wanted to understand how pilots behaved in a complex situation containing multiple incidents. This was carried out in a full motion simulator, and a complex scenario was played out which resulted in the co-pilot flying the plane, a number of go-arounds, a runway change, and low fuel throughout the scenario. From the other data (cockpit data, and voice recordings) and through expert analysis, it was clear there were pilots who dealt with the scenario more successfully than others. By using eye tracking outputs, we were able to develop theories as to why some pilots performed better than others, and attach metrics to the data, such as dwell times on specific instruments, or scan-paths which were repeated time and time again. We were then able to triangulate this with the cockpit transcripts and understand why certain things were looked at certain times, or indeed, allow us to explore later during interview why they weren’t.This information helped to inform more advanced cockpit instrumentation designs, as well as informing emergency training requirements. 

As a specific example of the insights afforded by Eye Movement Tracking (EMT), it was possible to see how the Captain and First Officer were cross-checking each other during the escalating emergency scenario, and how they supported each other. Typically, the FO, whose main fixation was the PFD (Primary Flight Display)  was able to ‘offload’ a degree of the situation awareness to the Captain so that he could focus on diagnosing the problems affecting their flight. EMT was also able to clearly pinpoint the moment when each crew realised they were low on fuel, and how they used various cockpit instruments to analyse an electrical failure. In some areas instruments such as the PFD were fully supporting the flight crew, whereas in others there was clearly room for improvement.

Maritime Domain:
In this study, navigator’s eye movement on the navigation bridge simulator was analysed during passage through the Istanbul Strait. A measurement device called “EMR-8” was used for eye movement tracking. In the application, three examinee groups were selected and they divided according to their onboard experience level. Each group has 4 examinees in real-time bridge simulator. Group 1 consisted of deck department students who had up to 2.5 months sea experience onboard ship as a cadet. Group 2 included junior officers who had already completed 12 months training on-board ship. Group 3 consisted of oceangoing masters who had wide experiences at sea. 

The scenario about passing Istanbul Strait was given to each examinee. The visual field was divided into three parts; inside, outside and others. The inside part has three components; instruments, indicators and an engine telegraph. The outside part has three components; the sea condition, navigational equipment and target ships. The scenario was recorded by eye movement tracking. In the light of findings, Group1 (beginners), gave minimum attention to navigation euqipments, target ships, regulations and sailing routes. Group 2, the examinees had intermediate characteristics as a navigation officer between professionals and beginners. They had enough knowledge and ability to use navigational equipments, target ships, regulations and routes. They had a longer duration on the bridge compared to Group 1. On the other hand, Group 3 had high level experience and knowledge. The examinees gave utmost attention to navigational equipments, sailing routes, target ships and regulations. Group 3 could stand on the bridge for the longest time without losing concentration. 

Key Concepts

The NEUROID is based on data-mining and machine-learning algorithms to make the HFs neurophysiological models fitting each operator and minimize inter-user variability, therefore obtaining reliable operator’s assessment. More than 88% of all general aviation accidents are attributed to human error. Therefore, NEUROID generates an innovative and systematic approach to quantify and objectively measure HFs by taking into account, at the same time, the behaviours, emotions, and the mental reactions of the operators themselves, and integrating them with the data related to accidents and incidents investigations. 

The general concept at the base of NEUROID is that brain, body, and operator’s experience are reciprocally coupled, and that accurate assessment of the operator’s can only be achieved through a well-defined combination of all the available data. Each biological activity is regulated by the human Nervous System, therefore variations of such biological activities correspond to internal reactions because of modification of external (environment) and internal (mental, motivations, emotions, etc.) factors. There are numerous types of neurophysiological measures, such as Electroencephalogram (EEG, related to brain activity), Electrocardiogram (ECG, hearth activity), Electrooculogram (EOG, ocular activity), Galvanic Skin Response (GSR, skin sweating), and so on. Such neurophysiological measures can be seen as the physical interface of the NEUROID technique that will enable to gather insights about all the aspects relating to HFs of the operator like Mental Workload, Stress, Vigilance, and Engagement. The key concept of NEUROID is the Human Performance Envelope (HPE) taxonomy: instead of considering one or two single human factors, the HPE investigates a set of interdependent factors, working alone or in combination, which allows to completely characterize the operator. In other words, these concepts are proposed as performance shaping factors, which can differentially and interactively affect successful completion of a task. The HPE theory explicitly declares that boundaries exist where performance can degrade in line with the theoretical underpinnings for the considered HFs.

Benefits

Traditional methods to catch information about the operators’ HFs are usually based on self-reports and interviews. However, it has been widely demonstrated how such kind of measures could suffer of poor resolution due to a high intra- and inter-user variability depending on the nature of the measure itself (i.e. subjective). In addition, the main limitation in using subjective and behavioural measures alone is due to the impossibility of quantifying ‘‘unconscious’’ phenomena and feelings underlying human behaviours, and most importantly it is not possible to measure such unconscious reactions experienced by operators while performing working activities. In fact, the execution of a task is generally interrupted to collect subjective evaluations. 

On the contrary, neurophysiological measures exhibit an unobtrusive and implicit way to determine the operator’s affective and cognitive mental states on the basis of mind-body relations. Therefore, the benefit and advantage of the NEUROID technique is to use neurophysiological measures and machine-learning algorithms to overcome such limitations, especially to (1) objectively assess the operator’s mental states while dealing with operative tasks; (2) identify the most critical and complex conditions and correlated them with accident and incident investigations; and (3) create a closed-loop between the systems and operators (i.e. Joint Human Machine Cognitive system) to continuously and non-invasively monitor the operators themselves.

Finally, modern wearable technology can help in overcome the invasiveness (e.g. many cables, uncomfortable) of the equipment necessary to collect the neurophysiological signals from the operators.

How It Works

• NEUROID consists in 3 main phases as reported in the figure. 

• The neurophysiological signals are gather from the operators by wearable technology while dealing with working tasks. 

• Afterward, the neurophysiological data are processed through a series mathematical steps towards the machine-learning phase, in which each HF is classified. 

• Finally, the considered HFs are combined to define the Operator’s HPE, and integrated with the operator’s behaviour, and data related to accidents and incidents (e.g. SHIELD database and HURID framework) for 

  a comprehensive operator’s assessment which can be employed for different applications, for example real-time mental states assessment, or triggering adaptive automations.

Illustrative Example

Aviation Domain
To see how the NEUROID technique can be implemented in practice we can consider the following scenario where an Air Traffic Controller (ATCO) is managing the air traffic and at the same time we are acquiring the ATCO’s EEG, ECG, and GSR neurophysiological signals. In the first phase of the scenario, the ATCO can rely on the support of high-automated systems to manage high traffic situation (High). Then, the traffic demand returns to a normal condition (Baseline), but at a certain moment the automations crash (Malfunction) and the ATCO will have to keep managing the traffic, and find out what is wrong with the automations.

In this context, we can use the NEUROID to measure how the ATCO’s HPE changes. In this regard, the neurophysiological data will be employed to estimate the ATCO’s Mental Workload, Stress, Attention, Vigilance, And Cognitive Control Behaviour based on the S-R-K model. Finally, those HFs will be combined to define the ATCO’s HPE on the three operational conditions as shown in the following figure. In particular, the values of the considered mental states have been normalized within the [0 ÷ 1] range: “0” means Low, while “1” mean High. Concerning the S-R-K aspect, “0” represent the S (Skill), “0.5” the R (Rule), and “1” the K (Knowledge) level.
 

The analysis of the HPE in the different scenario conditions exhibits interesting variations on the configuration of the mental states. In fact, the failure of the automations (red line) induced higher Vigilance, lower Attention, and shift from Skill to Rule level in the Cognitive Control Behaviour with respect to the HIGH (orange line) and BASELINE (blue line) conditions. Furthermore, considering the HPE changes over time throughout the ATM scenario allows to identify those moments, thus situations, corresponding to high (RICH) or low (POOR) performance. For example, for the considered ATCO, the HPE corresponding to RICH (green plot) and POOR (red plot) performance are reported in the following figure. Generally, in correspondence of RICH performance condition, the vigilance is higher and the stress is lower than during POOR performance. In conclusion, the NEUROID could be used, for example, to monitor the operators and warn them when they are working under low vigilance and high stress condition.

Maritime Domain
To see how the NEUROID technique can be implemented in Maritime practice we can consider the following scenario where the collision occurred between tanker ship and livestock ship during Dardanelle Strait passage at sea. The collision occurred during overtaking. The VTS (Vessel traffic services) was informed about the collision since it occurred during Dardanelle Strait passage. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer was keeping the navigational watch at the time of collision. It was early morning at the time of collision.

The HF analysis of the ship collision during Strait passage would highlight the following potential root causes: 

• Mental Fatigue: Rather than physical fatigue, over alertness and overload (decision making, concentration, mental load). Watchkeeper Officers continuously working load can affect his/her decisions.

• Physical Fatigue: Rather than mental fatigue, physical fatigue can be seen due to intensive work load on-board ship. 

• Inadequate Manning on Bridge: According to sailing zone, increase the number of watchmen on bridge may help perception of the dangers earlier.

• Inadequate Policy, Standards and Application: Increase of the inspections due to sailing rules and minimum CPA-TCPA values and activation of the existing rules may help avoiding dangerous situations and provide clearance from the plotted vessels.

• Inadequate / Lack of Communication: Necessary manoeuvre could not be applied due to inadequate communications between vessels. Lack of communications between vessels and VTS also caused vessels to sail in close range. Every communication methods on board must be managed in most efficient manner.

• Poor Risk Evaluation: Main reasons of the incident between vessels are the failure of the evaluation of risks for all vessels in the sailing area and unforeseen of the other vessel possible manoeuvres which were sailing in the outmost line of the separation. 

• Inadequate Leadership and Supervision / Planning: Another root causes of the incident are improper planning of watch hours and not taking necessary measurements as a result of evaluations of inconveniences timely.

• External/Environmental Conditions: Sailing in narrow zone, congestion and poorness of the VTS stations in planning the traffic are the main effective reasons of the incident. To prevent such like incidents, environmental conditions must be evaluated sufficiently and proper speed and watch order must be planned according to congestion.

Some of those aspects are mainly linked to the management level (e.g. manoeuvres procedures, risk evaluation), but the ones coming from the evaluation of the operators’ status like the mental and physical fatigue can be monitored and measured via the NEUROID during the working activities. 

In particular, along each phase of the considered scenario the neurophysiological measures can characterise the operators in terms of Mental Workload, Stress, Vigilance, and Engagement combination and those information can be consequently used to:

• interact and\or intervene in real-time on the system, for example depending on the Master's mental workload it could be possible to enable a warning during overload or low vigilance conditions to regain the proper operational status;

• post-doc analyse the incident\accident or simulation by combining the previous tasks analysis, and the operators' mental states profile along the scenario, for example to find out how the Mental Workload, Stress, Vigilance, and Engagement of the Master and C/O were right before the collision or during a specific phase which brought to the collision, and finally understand if the configuration of those mental states was not appropriate (e.g. distracted?) or, on the contrary, the operators were under too high demanding and stressing conditions due to the adopted procedure or situation.

Key Concepts

The Systemic Occurrence Analysis Method is one of several accident analysis tools based on principles of the well-known "Swiss Cheese Model" of organisational accidents. SOAM is a process for conducting a systemic analysis of the data collected in a safety occurrence investigation, and for summarising this information using a structured framework and standard terminology. As with some root-cause analysis investigation methods, SOAM draws on the theoretical concepts inherent in the Reason Model, but also provides a practical tool for analysing and depicting the inter-relationships between all contributing factors in a safety occurrence. SOAM allows the investigator to overcome one of the key historical limitations of safety investigation - the tendency to focus primarily on identifying the errors - those intentional or unintentional acts committed by operators - that lead to a safety occurrence. Reason's original model has been adapted and refined within SOAM. The nomenclature has been altered in accordance with a "Just Culture" philosophy, reducing the implication of culpability and blame by both individuals and organisations. In SOAM, 'Unsafe Acts' are referred to as Human Involvement, 'Psychological Precursors of Unsafe Acts' as Contextual Conditions, and 'Fallible Decisions' as Organisational and System Factors.

Benefits

SOAM forces the investigation to go deeper than a factual report that simply answers questions such as “What happened, where and when?" First, data must be collected about the conditions that existed at the time of the occurrence which influenced the actions of the individuals involved. These in turn must be explained by asking what part the organisation played in creating these conditions, or allowing them to exist, thereby increasing the likelihood of a safety occurrence. SOAM thus supports the fundamental purpose of a safety investigation - to identify and understand the factors that contributed to an occurrence and to prevent it from happening again.

SOAM is aligned with, and supports, "Just Culture" principles by adopting a systemic approach, which does not focus on individual error, either at the workplace or at management level. It avoids attributing blame by:

• Removing the focus from people's actions, and instead seeking explanation for the conditions that shaped their behaviour;
• Identifying latent organisational factors that allowed less than ideal conditions to exist, under which a safety occurrence could be triggered.

As with the original Reason’s Model, SOAM can be applied both reactively and proactively. The process can be applied to any new occurrence, and is also suitable for the retrospective analysis of previously investigated occurrences in an attempt to extract additional learning for the promotion of safety. SOAM can also be applied proactively to generic occurrences or hypothetical events. These applications result in a comprehensive analysis of the absent or failed barriers and latent conditions that are commonly found to contribute to such events, thereby identifying areas of organisational weakness that need to be strengthened to improve safety and prevent future occurrences.

How It Works

The SOAM process follows seven specific steps that are Gathering data, identifying barriers, Identifying human involvement, Identifying contextual conditions, Identifying organizational factors, Prepare SOAM chart and Formulate recommendations.

GATHERING DATA:
While there is no definitive or prescribed method for the gathering of investigation data, it is useful to gather data within some form of broad descriptive framework, to help with the initial sorting of facts. The SHEL Model provides the basis for such a descriptive framework.
Data should be gathered across five areas (the four original areas of the SHEL model, and an extra fifth element - organisation):

• Liveware
• Software
• Hardware
• Environment
• Organisation

While the data gathering and analysis phases in an investigation are typically depicted as discrete, in reality they are part of a recursive process. After an initial data collection phase, a preliminary analysis can be conducted, which will identify gaps that can be filled by further data gathering. This process will continue until the systemic analysis has eliminated unanswered questions and reached a logical conclusion.

Having collected the data, the first stage of the SOAM analysis involves sorting each piece of factual information into an appropriate classification. This is a progressive sorting activity which can be conducted as a group exercise if the investigation is being conducted by a team.

IDENTIFYING ABSENT/FAILED BARRIERS:
Barriers protect complex socio-technical systems against both technical and human failures. Absent or failed barriers are the last minute measures which failed or were missing, and therefore did not (a) prevent an action from being carried out or an event from taking place; or (b) prevent or lessen the impact of the consequences.
As with each stage of the SOAM process, a check question is applied to ensure that the item being considered fits within the definition of the category it is being considered for. 

IDENTIFYING HUMAN INVOLVEMENT:
Following identification of the relevant absent or failed barriers, the next step is to identify the contributing human actions or non-actions that immediately preceded the safety occurrence. The question at this stage should not be why people behaved as they did, but simply what their actions/inactions were just prior to the event. 

The information-processing model selected for use with this methodology is Rasmussen's Decision Ladder technique (1982). Like similar models, it assumes that information is processed in stages, beginning with the detection of information and ending with the execution of an action.

The Decision Ladder uses a six-step sequence which has been adapted for use within SOAM to present a simplified view of common controller tasks such as ATCO. Using this model, human involvement in a safety occurrence can be analysed in terms of:

• Observation 
• Interpretation
• Choice of goal
• Strategy development
• Choice of action plan
• Execution of action plan

IDENTIFYING CONTEXTUAL CONDITIONS:
Contextual conditions describe the circumstances that exist at the time of the safety occurrence that can directly influence human performance in the workplace. These are the conditions that promote the occurrence of errors and violations. In the occurrence investigation process, contextual conditions can be identified by asking “What were the conditions in place at the time of the safety occurrence that help explain why a person acted as they did?"
Five categories of contextual conditions can be distinguished, two relating to the local workplace, and three to people:

• Workplace conditions
• Organisational climate
• Attitudes and personality
• Human performance limitations
• Physiological and emotional factors

IDENTIFYING ORGANISATIONAL FACTORS:
The organisational factors (ORFs) describe circumstances which existed prior to the occurrence and produced or allowed the continued existence of contextual conditions, which in turn influenced the actions and/or inactions of staff. The following categories of ORFs are identified:

• TR - Training
• WM - Workforce Management
• AC - Accountability
• CO - Communication
• OC - Organisational Culture
• CG - Competing Goals
• PP - Policies and Procedures
• MM - Maintenance Management
• EI - Equipment and Infrastructure
• RM - Risk Management
• CM - Change Management
• EE - External Environment

THE SOAM CHART:
The final product of the systemic occurrence analysis process is a summary chart depicting:
• The individual contributing factors - grouped according to the layers of the methodology as Barriers, Human Involvement, Contextual Conditions and Organisational Factors; and
• Horizontal links representing the association between a contributing factor at one level (e.g., a human action), and its antecedent conditions (i.e., the context in which the action took place).

FORMULATING RECOMMENDATIONS:
The formulation of recommendations for corrective action is a critical final element of the occurrence investigation process. In order to be effective, the recommendations should have the following characteristics:

• Be directly and clearly linked to the SOAM analysis;
• Be focussed on findings that are amenable to corrective action;
• Aiming to reduce the likelihood of a re-occurrence of the event, and/or reduce risk.

In formulating recommendations, the SOAM process requires that the following two elements be addressed:

• The deficient Barriers (absent or failed), and
• The Organisational Factors.

Illustrative Example

Aircraft runway collision
The Linate Airport disaster occurred on 8 October 2001 at Linate Airport in Milan, Italy, when Scandinavian Airlines Flight 686, a McDonnell Douglas MD-87 airliner carrying 110 people bound for Copenhagen, Denmark, collided on take-off with a Cessna Citation CJ2 business jet carrying four people bound for Paris, France. All 114 people on both aircraft were killed, as well as four people on the ground. The subsequent investigation determined that the collision was caused by a number of non-functioning and non-conforming safety systems, standards, and procedures at the airport. It remains the deadliest accident in Italian aviation history.

The SOAM Chart of the Milan Linate accident highlights the role played by a number of latent failures in the chain of events that brought to the disaster. For example, in the analysis the boxes highlighted in blue show the path of contextual conditions, organizational factors and other system factors that acted as precursors of the error committed by the flight crew of the Cessna aircraft causing the runway incursion.

Illustrative Example: Collision between Hampoel and Atlantic Mermaid
On 7 June 2001, the Panamanian- registered refrigerated cargo vessel Atlantic Mermaid, collided with the Cypriot-registered general cargo vessel Hampoel, off the Varne in the south-west bound lane of the Dover Strait traffic separation scheme (TSS). An investigation of the accident was conducted by the UK’s Marine Accident Investigation Branch (MAIB). 

The SOAM Chart of collision between Hampoel and Atlantic Mermaid highlights the role played by a number of failures in the chain of events that brought to the accident. The SOAM Chart was developed based on the identified root caused and findings obtained from the investigation.

Key Concepts

STAMP is a systemic method, which means that’s strong links exist between the different components of the system that directly influence each other. In other words, the STAMP method views systems as components that are interrelated in a state of dynamic equilibrium.

As a systemic method, it takes effort to apply as it requires a deeper analysis of the regular processes and organisation in order to map them on system-theoretical feedback-control loops.

STAMP uses a System-theoretical control cycle model, containing the process under control, sensors, actuators, controllers and conceptual models governing the decision taken to control the process.

STAMP is similar to FRAM in the way in which they link different parts of the system by acknowledging the tight coupling of individual functions and constituents of the system.

Within STAMP method, accidents are the result of errors from interaction among people, organizational structures, engineering activities, and systems components. Hence, an accident occurs when an adaptive feedback function fails to maintain safety and performance over time.

STAMP treats safety and resilience as control problems, in which accidents occur when component failures, external disturbances or interaction between system components are not adequately addressed.

The diagram below shows a control cycle in which a person is controlling the process of flying a drone. This involves receiving feedback about whether the drone is behaving as intended and implementing adaptive control actions when needed. 

Benefits

On the positive side, STAMP presents a long list of benefits:

• It provides an overview of all the links that exist between all the components being modeled and their interactions.
• It is appropriate for complex, tightly coupled systems.
• It has a good track record in several industries (e.g. aviation or maritime).
• The socio-technical control model of STAMP integrates both, the system development (e.g. legislation, design or implementation) and the system operations (e.g. legislation or operating process).
• It is versatile, and can be applied to all sorts of systems. Specifically, STAMP is a very adequate method for analysing an accident in any technical system.
• It can be used early on (i.e. at the design stage).

On the negative side, STAMP also presents several well-known disadvantages:

• As a systemic method, STAMP is resource-intensive (i.e. an analysis using a sequential method rather than a systemic method may be easier to understand, communicate, and quicker to execute).
• It requires a high level of expertise.

How It Works

STAMP is designed around three major areas, in which each area allows classifying certain controlling errors that could potentially lead to an accident. Thus, these areas have been defined by Erik Hollnagel et al. (2007) as follows:

Finally, the STAMP method involves creating a model of the organizational safety structure. This model can be applied to investigate incidents or accidents, aiming to establish the role played by any components regarding the safety control structure. Thus, the above-mentioned model can also be utilized to learn how to prevent a future accident from taking place, to perform hazard analysis and reduce risks, or to create and support a risk management program in which risk can be controlled and monitored.

Illustrative Example

Aviation Domain: Loss of a U.S. Army Black Hawk helicopter from friendly fire.

Accident Summary 
On April 15, 1994, two US Air Force F-15 shot down two US Army Black Hawn helicopters, mistaking them for Iraqi Hind helicopters, killing 26 people. The aircraft were flying in clear weather with excellent visibility.

Hierarchical Safety Control Structure: 
Understanding why this accident occurred requires understanding why the control structure in place was ineffective.

Constraints and Control from each safety control structure: this step is conducted at each level of the hierarchical safety control structure. An example is included below for the Mission Director (MD) and Airborne Command Element (ACE).

Accident Analysis: Using STAMP to understand why this accident occurred and to learn how to prevent it in the future requires determining why the safeguards were not successful in preventing the friendly fire.

Step 1: Physical failures and dysfunctional interactions. 

The first step in the analysis is to understand the physical failures and dysfunctional interactions within the physical process that were related to the accidents. This step is conducted at each level of the hierarchical safety control structure. An example is included for the MD and the ACE.

Context in which Decision and Actions Took Place.

Dysfunctional Interactions at This Level: The ACE had to get information about enemy aircraft but it was not provided.

Flawed or Inadequate Decision and Control Actions: The ACE did not provide any control commands to the F-15.

Reasons for Flawed Control Actions and Dysfunctional Interactions.

Changes after the Accident: There were not changes but roles were clarified.

Step 2: Conclusions from STAMP analysis. While there were mistakes made at the pilot and AWACS levels, the use of the STAMP analysis paints a much more complete explanation, including:

• Inconsistency, missing or inaccurate information.
• Incompatible technology.
• Inadequate coordination and training.
• Overlapping areas of control and confusion about who was responsible for what.
• A migration towards more efficient but less safe procedures over time without checks on the potential adaptations.
• Moreover, a control structure that did not enforce the safety constraints.

Maritime Domain: Accident of the MV Herald of Free Enterprise. 

Accident Summary: On the evening of the 6th of March, 1987, the Roll on/Roll off Ferry Herald of Free Enterprise left the berth in the port of Zeebrugge at 18.05. About 20 minutes after her departure, the ferry passed the outer mole and capsized minutes later. 150 passengers and 38 crew members died in the accident

Hierarchical Safety Control Structure: Understanding why this accident occurred requires understanding why the control structure in place was ineffective. For this STAMP analysis, information was obtained from the following sources: i) the original accident report; and ii) research papers on the accident.

Constrains and Control from each safety control structure: This step is conducted at each level of the hierarchical safety control structure, starting from the bottom. An example is included for the Master of the MV Herald of Free Enterprise.

Accident Analysis: Using STAMP to understand why this accident occurred and to learn how to prevent it in the future requires determining why the safeguards were not successful in preventing the sink of the MV Herald of Free Enterprise.

Step 1: Physical failures and dysfunctional interactions. The first step in the analysis is to understand the physical failures and dysfunctional interactions within the physical process that were related to the accidents. This step is conducted at each level of the hierarchical safety control structure, starting from the bottom. An example is included below for the Master of the MV Herald of Free Enterprise.

Context in which Decision and Actions Took Place.

Dysfunctional Interactions at This Level: The Master who served on board relied upon the absence of a report telling him the ship was not seaworthy.

Flawed or Inadequate Decision and Control Actions: The Master took the vessel to sea with the bow doors open.

Reasons for Flawed Control Actions and Dysfunctional Interactions.

Changes after the Accident: No changes were made at this hierarchical safety level.

Step 2: Conclusions from STAMP analysis. After conducting a physical failures and dysfunctional interactions analysis at each level of the hierarchical safety control structure, a set of conclusion can be extracted. While there were mistakes made by the crew members, the use of the STAMP analysis paints a much more complete explanation, including:

• The vessel capsized because she went to sea with her bow doors open allowing water to ingress on the car deck.
• Inadequate communication and co-ordination between crewmembers.
• The capsizing was partly caused or contributed to by serious negligence of three crew members (Master, Chief Officer and Assistant Bosun) and partly caused or contributed to by the fault of the Owners.
• Overlapping areas of control and confusion about who was responsible for what.
• The Master who served on board relied upon the absence of a report telling him the ship was not seaworthy. This was a dangerous assumption, which leads to him taking the Herald of Free Enterprise to sea with the bow doors open.
• The underlying faults lay higher up in the Company. The company gave evidence of not realizing their responsibility for the safe management of their ships.

Key Concepts

A view is governed by its viewpoint: the viewpoint establishes the conventions for constructing, interpreting and analysing the view to address concerns framed by that viewpoint. These views are:

The separation of the whole system of interest into different viewpoints enables a focus on specific concerns while hiding other possibly irrelevant details. As such the user (e.g. a software developer, an analyst or end-user) can zoom in on specific aspects while disregard other aspects. And, as the different viewpoints in union compose a more complete system the user can zoom out and get a more holistic view on the system.

Benefits

On the positive side, ARKTRANS has over a number of years been used in the development of Enterprise Architectures related to the transport domain as well as in the aviation. It is typically applied on complex environments or systems. 

On the negative side, different viewpoints may use different notation, but often UML is used in some variant. It is the case of ARKTRANS which uses UML consistently as its architectural notation. The roles and the functional viewpoint prescribe the use of UML Use Case notation; the process viewpoint uses UML activity diagrams; and the information viewpoint applies UML class diagrams as the notation for describing information elements. 

In a recent work, ARKTRANS has been improved using BPMN to model processes whereas ARKTRANS uses UML Activity diagrams. The notation is pretty similar, BPMN is more expressive with its extended use of stereotypes and especially the possibility to express different activity types (e.g. in BPMN one can clearly state through symbols and tagged values that an activity is performed by a human or a technical system). This new framework is also supported by Enterprise Architect tool.

How It Works

It prescribes a top-down approach, starting with conceptual aspects at the top, working its way down to logical aspects and finally down to technological aspects. The current standard prescribes how to develop enterprise architecture frameworks and the manner in which architecture descriptions of systems are organized and expressed. 

Further, it specifies architecture frameworks defined as conventions, principles and practices for the description of architectures established within a specific domain of application and/or community of stakeholders and architecture viewpoints defined as a work product expressing the architecture of a system from the perspective of specific system concerns for use in architecture descriptions.

Illustrative Example

Aviation Domain
Go-around procedure and the Arlanda case study
This illustrative example is taken from SCALES project which also enhanced ARKTRANS and its conventional viewpoints by adding the resilience one.

A Go-around (G/A) is an aborted landing of an aircraft that is on final approach. It can be adopted when everything is not quite right for landing. Initiation of a G/A procedure may be either ordered by the controller or decided by the pilot in command for several reasons. A G/A does not in itself constitute any sort of emergency. Indeed, a properly executed G/A is a safe, and well-practiced manoeuvre; however it requires proper execution by pilot and adequate support by the controller. It occurs with an average rate of 1-3 per 1000 approaches and one in ten go-around lead to a potentially hazardous outcome. Several elements play a role during a G/A such as, pilots, aircraft elements used for landing and taking off, runway and weather conditions, ATCOs supervise the landing part of the flight, landing systems used by both pilots and ATCOs, safety nets as the Ground Proximity Warning System, communication devices between pilots and ATCOs, procedures for landing, runway layout, managing G/A and re-routing of an aircraft.

In a resilience perspective, a G/A can be considered as an event which generates a disturbance into the system under analysis (e.g. an ACC) which has to deal with it. How the ACC copes with this disturbance allow us to identify some contributing factors, positive and/or negative, which can be correlated to resilience aspects.

SCENARIO: On January 21, 2011, an aircraft aborted at a late stage the landing on runway 26 at Arlanda airport at the same time as another aircraft took off from runway 19R. At missed approach to runway 26 right turn should be carried out as soon as possible in order to avoid conflict with departing traffic from another runway. The aircraft that aborted the landing followed the prescribed procedures for missed approach only after three calls from the ATCO. Although the conflict was observed by the ATCO and the departing aircraft had been instructed to change its course a separation infringement occurred. During the G/A, the Commander took over the control of the aircraft. The Commander’s taking over the controls at a late stage probably resulted in a lack of sufficient remaining capacity to immediately follow the published missed approach procedure

The first step of ARKTRANS is performed without any architectural support and includes the identification of possible contributing factors from the incident report (as reported in the table). Then, the roles, functions, activities and information exchanges related to each of the identified contributing factors are described in a matrix form. 

For example, the contributing factor “The Commander experienced information overflow when receiving information from traffic control and aircraft instruments in an already stressful situation” can be described as reported in table below. The table exemplifies that for each identified contributing factor the architectural constructs are mapped. In the “Involved Role(s) and actors” column the role includes the actor (i.e. the instantiation of the role in this particular case) in parenthesis. In this case the activities mentioned are equivalent with the functions, but this is not necessarily so. An activity can also be a sub-part of a more complete function (i.e. the relevant responsibility of the role). In this particular example there were 3 ATC instruction messages sent from the tower controller to the aircraft but only the third was acknowledged by the pilot. The table also presents ATC roles and prompt ATCOs interaction.

So, the role, the functional, the information and the process views for the specific case study are developed (see the figure below). Using existing constructs from the baseline architecture, the process view for this particular case is developed according to the sequence of events. From the baseline architecture we can say that the roles, functional view and information view constitute a knowledge base or a catalogue from which inherent constructs can be collected from when constructing the process view. That is, the roles, the functional view and the information view are more static representations across the cases. If a case introduces new roles (or actors), new functions or new transactions these views are updated in the framework (due to space constraints, an excerpt of the Arlanda case study process view is provided in the figure below). The reason for this is that although the roles, functions and transactions are important, the relevant constructs from these are also included in the process view and as such this is the view that provides the most detail.

Once having developed the process view, it is possible to extract candidate resilience indicators including sequences of activities as well as interaction and information exchange between roles and actors. For this we are applying UML state diagrams. The intention of a state diagram is to represent states and their triggers. Considering the nature of resilience, we are interested in a system state (either positive or negative) and the triggering factors that bring the system into such a state. From this, state diagrams enable us to “zoom in” on the details regarded as most relevant and abstract away details that are less relevant and that potentially can blur the overall picture. The rounded rectangles indicate the main states while the arrows between them are their triggers. Extracted from the triggers are candidate resilience indicators which enable the consolidation of resilience patterns. The consolidation is a composition of resilience patterns and associated candidate resilience indicators forming such patterns. The «Resilience Pattern» is the “Organization of Traffic management TWR” which is constituted by 2 «Resilience indicators» that are “Information exchange between aircraft and ATC – Timing and Synchronization” and “Controller-Controller actively monitoring each other”.

Maritime Domain 

The case of ship Collision
This illustrative example is taken from accident report related with ship collision during strait passage.  

SCENARIO: The collision occurred between tanker ship and livestock ship during Dardanelle Strait passage at sea. The reason of the ship collision was not to follow up COLREG (collision regulation) rules during overtaking. The VTS (Vessel traffic services) was informed about the collision since it occurred during Dardanelle Strait passage. The weather was partly cloudy and sea state was calm at the time of collision. It was early morning at the time of collision. 

At the result of the examinations, the tanker ship which were steering with 11,7 knot speed to the course of 244° in Marmara Sea in the direction of Dardanelles, have contacted with her starboard bow to port stern quarter of the livestock ship within 20 nm range of Dardanelles at 06:25 LT on the date of 01.10.2013. Development of the event was as follows; tanker ship overtook another ship when she was steering close to the middle of the separation in the direction of Dardanelles. It has been observed that livestock ship were steering in the starboard bow of other ship in direction of Dardanelles and close to outmost line of the separation where she was also in the range of about 0,6 NM with tanker ship. It also has been observed that other vessel was steering in the starboard bow of tanker ship within the range of 1 mile. 

At the result of investigation of ECDIS records and statements of the officer on watch, it has been perceived that; After taking over the other ship with the range of 0,3 NM and heading towards to other ship, tanker ship have contacted with livestock which were steering outmost line of the separation. Even though small course alteration to port side that was made by tanker ship, contact couldn’t be prevented due to sudden and big course alteration to the port side had been made by livestock ship.

Key Concepts

Safety culture has a number of dimensions, such as the following (Kirwan et al, 2016):

• Management commitment to safety (extent to which management prioritize safety);
• Communication (extent to which safety-related issues are communicated to staff, and people ‘speak up for safety’);
• Collaboration and involvement (group involvement and attitudes for safety management);
• Learning culture (extent to which staff are able to learn about safety-related incidents);
• Just culture and reporting (extent to which respondents feel they are treated in a just and fair manner when reporting incidents);
• Risk handling (how risk is handled in the organisation);
• Colleague commitment to safety (beliefs about the reliability of colleagues’ safety behaviour);
• Staff and equipment (extent to which the available staff and equipment are sufficient for the safe development of work);
• Procedures and training (extent to which the available procedures and training are sufficient for the safe development of work).

Safety culture is typically evaluated using an online questionnaire of validated questions that is distributed across the organisation and filled in anonymously (usually taking about 10 minutes). Based on the responses, analysts then identify the safety culture ‘hotspots’, which are then discussed in confidential workshops, leading to recommendations for the organisation to improve its safety culture. At a macro level, typically a ‘spider diagram’ is produced based on the results, highlighting which safety culture dimensions need most attention. In the example figure to the right, ‘Colleague commitment to safety’ is doing well in safety culture terms, whereas ‘Staffing and Equipment’ requires attention.

Benefits

Safety culture surveys, particularly when anonymous and carried out by an independent organisation, are often seen as one of the only ways to find out all your risks and what people really think about the state of safety and safety culture in an organisation. They can pinpoint both current issues that can be corrected (so-called ‘quick wins’) without too much trouble, as well as deeper problems that may be entrenched in the organisational culture. Most safety culture surveys do identify the safety culture strengths of an organisation as well as vulnerabilities, so there is always some ‘good news’ as well as areas for improvement. Because the approach uses workshops of operational personnel, usually the recommendations that arise from surveys are useful and practicable, since they come from the organisation itself. 

The survey process is long, however, often taking 9 months from the first decision to have a survey, to having a report with recommendations. The management need to be committed to doing the process and realise that it is not a quick fix, and there will be work to do afterwards. If management are doing the survey just to gain ‘a tick in the box’, then it may be better not to do one at all.

How It Works

In many application of the aviation domain, there are two main phases used to evaluate the safety culture in a safety culture survey:

Phase 1: Questionnaire 
The safety culture survey uses an electronic version of the Safety Culture Questionnaire, over a period of typically 3 weeks. The confidential data are anonymised by The London School of Economics and then independently analysed by EUROCONTROL

Phase 2: (Optional) Workshops 
After an initial analysis of the questionnaire results, there is a visit to the organisation’s premises to hold a number of confidential workshops with front-line staff (without managers present) to clarify and explore further certain aspects emerging from the questionnaires. An example of what might be discussed in such a workshop is indicated by the figure below, which shows results for an organisation in the safety culture dimension ‘Staff and Equipment’. With respect to the questionnaire statement ‘We have the equipment needed to do our work safely’, whilst more than 70% were in agreement (the green bar), the rest were either unsure or in disagreement. The workshop would explore equipment issues or concerns people have, which are underlying this response.

Illustrative Example

Aviation domain
It should be noted that the example reported was seen as an exploratory survey and were focused on a single particular segment of the organization – Airbus Design rather than being company-wide (see Kirwan et al, 2019).

For the Airbus survey, the focus was on the Design part of the organization, which involves around 2,000 of the 70,000 employees. This survey required considerable tailoring of the questionnaire, as the job of design and systems engineering is somewhat different from airline operations, even if most underlying issues, except fatigue, and dimensions remain relevant. But the tailoring worked, with the same issues raised during workshops as via the questionnaires. Participation in the workshops by designers from all three of Airbus’ main design locations (France, UK, and Germany) was intense. The general principle espoused in each workshop was that a safety defect in a design of an aircraft could simply not be allowed to happen; therefore, any safety issues had to be resolved carefully and thoroughly before continuing, even if resolution efforts delayed progress toward production. One observation from the study team was that the managers in the management workshop were connected with safety and detailed safety concerns, more so than is often the case for middle managers. This is because many of the managers had held operational positions or positions ‘close to operations,’, so that they understood the operational pressures that could affect airlines and pilots. Another had been to an accident site after an air crash, and noted that after such an experience ‘safety never leaves you.’

The workshops focused a lot on internal communications between the different design departments, as well as the resolution of safety issues that could delay production, and the importance of middle manager support during such resolution periods. 

A number of constructive recommendations were made to Airbus in the final survey report.
 

Maritime domain
A safety culture assessment and an implementation framework were developed by the University of Strathclyde to enhance maritime safety. The focus was on the development of a safety culture assessment tool which covers all of the safety related aspects in a shipping company and measures will be taken proactively and reactively in order to enhance safety of the shipping industry.

Questionnaires were developed with an interdisciplinary group to ensure that they capture the fight information to conduct a comprehensive analysis. Each safety factor has specific questions which try to address the employee’s attitudes and perceptions amongst crew members and shore staff. There are a total of 85 questions which are asked to the employees, under the headings of communication, employer-employee trust, feedback, involvement, problem identification, promotion of safety, responsiveness, safety awareness and training and competence. 

The aim of the surveys is to analyse strengths and weaknesses in the company and perform benchmarking between crew and shore staff’s attitudes towards safety. Participants of the surveys were asked questions about demography, safety factors and open-ended questions about company policies. The safety culture assessment framework requires commitment from all of the bodies in a shipping company for successful implementation. 

Based on the findings from the questionnaires a set of recommendations were provided to the shipping company to improve communication, employer-employee trust, feedback, involvement, mutual trust, problem identification, responsiveness, safety awareness, training and competence, and the promotion of safety.

Key Concepts

Actor: The actor is the user who experiences the journey. The actor is who the journey map is about — a point of view. The map can include such information as age, occupation, location, as well as details like what device they’re on and what task they want to accomplish. It is advisable to create user journey maps for each of the target users, as their goals and expectations could differ.

Scenario and expectations: The scenario describes the situation that the journey map addresses and is associated with an actor’s goal or need and specific expectations. 

Phases: Journey phases are the different high-level stages in the journey. They provide organization for the rest of the information in the journey map (actions, thoughts, and emotions).

Actions, Mindsets and Emotions: These are behaviours, thoughts, and feelings the actor has throughout the journey and that are mapped within each of the journey phases.

Insights: Opportunities (along with additional context such as ownership and metrics) are insights gained from mapping; they speak to how the user experience can be optimized. Insights and opportunities help the team draw knowledge from the map.

Benefits

1. On top of the Hierarchical Task Analysis it adds the affective and cognitive part. Rather than only focusing on the functional elements of the steps, it addresses emotion in design: it maps out the user’s expectations, questions, thought processes and feelings when using the product.

2. Through thoughts and emotions it is easy to see the problems of the users, which shows how the system should be further refined (i.e. opportunities), who is responsible for the development (i.e. internal ownership), and also allows for prioritisation within the team.

3. The process of creating a map forces conversation and an aligned mental model for the whole design team, which may even be more valuable than the finalised map itself. 

4. Understanding the user’s journey – their feelings, motivations and experiences – can help you design a user interface that guides them towards meeting their needs. 

How It Works

Think about the approach you want to take:

 - Do you want to focus on currently available product and illustrate the current problems end-users may encounter, or do you want to think about an ideal experience?

 - Do you already have the data from research, or do you need a solid foundation first that will be validated later with the user insights?

Draw the skeleton of the journey map (actor, scenario and expectations, phases, actions, thoughts, emotions, interaction, opportunities). Templates can be found online (see Kaplan, 2006), but there is no universally accepted version of the map; it boils down to the creativity of the team of how unique they want to make the map. The single most important thing is to illustrate all the key elements of the map described above. 

Once the skeleton is ready, the process starts with Zone A:

(1) First, agree on the actor(s) the team wants to cover. Everything that comes after has to be created from the actor’s perspective. As a rule, one actor is covered in one journey map.

(2) Scenario and expectations put the whole map into context. Note that even these two can alter between different actors, but only one scenario should be elaborated in one journey. 

The next part to be filled out is Zone B: 

(3) Think about the main phases the actor would go through during the journey. They should be worded as verbs to reflect what the user is doing in that set of actions, again from the user’s point of view.

(4) List the set of actions, thoughts (4) and emotions (5), categorized into the high-level journey phases. 

The last zone is Zone C:

(7) List the possibilities to improve the journey (opportunities) consequently, one can assign priorities to the functions to be developed. 

(8) Ownership and metrics can be added as well, the former facilitates the clarity and awareness of the internal responsibilities to develop the particular functions.

Illustrative Example

Aviation Domain: 

Unmanned Traffic Management (UTM) application prototype for mobile features

The journey mapping was applied in a case study that aimed to create a simple mobile application design for drone pilots within U-Space environment. The motivation came from the problem that Unmanned Aircraft Systems (UAS) operators could be using multiple applications to be able to conduct their flights, submit their flight plan and monitor the status of the airspace of the vicinity of their mission. The use of these applications in parallel may have a significant safety risk which has to be investigated to ensure the safe and efficient operations.

Prior to the journey mapping, the design team created a Hierarchical Task Analysis (HTA) and run couple of usability tests. Therefore the journey map was:

A mixture of future state and current state: Whilst during the activity of creating the journey map the designer team had existing problems with existing apps in mind (current state), this journey map represents a forward-looking scenario. New ways of interactions for emerging roles and responsibilities had to be defined. Therefore both the HTA and the subsequent journey map was a visualization of an ideal journey (future state).

Research-first: After the usability tests the team created the journey map from scratch, but the actions were taken from HTA and have been aligned with prototype. Since the team already knew the user insights by the time the map was created, the process can mainly be regarded as “research-first”- all the thoughts, emotions and opportunities come from real-life testing.

Based on the outcomes of the usability test it became evident that there were two types of users; 

  - one who clicked through the prototype fast and hasn’t given too much thought to minor details;

  - and the other who reads everything, analysed all the small details of the interface and had a lot of comments throughout the journey.

The apparent difference prompted the creation of two journey maps each describing one type of user. The two actors presented in the journey maps are called  Matt and Sophie, they are both drone pilots whose journey of using the UTM application during a specific event is shown below. Sophie belongs to the first group and has a rather intuitive personality, whilst Matt is more of an analytical type of user and who is very detail-oriented.

The event that is covered in the journey maps, describes how the two drone pilots execute a foreign object detection (FOD) mission on the runway, within the controlled traffic region (CTR). An arriving aircraft in emergency leads to the suspension of the drone activities in the CTR, thus the drone has to return home immediately. After a while the operation can be resumed and this is what  the scenario within both journey maps  is about. This scenario requires the drone pilot to create a new flight plan and fill out a pre-flight checklist prior to resuming the mission. To accomplish the goal, the drone pilot has to interact with two software: the app from the drone manufacturer to control the drone and the UTM app to file the flight plans, fill out checklist, monitor the airspace and keep contact with air traffic control (ATC). To sum up, the focus within the journey maps lies on the submission of the flight plans when resuming the FOD mission. 

The user journey maps have been created in MS Visio. The design team that took part in the usability test as observes was split into two groups – Team Matt and Team Sophie. Each team illustrated the journey for the according actor. 

The two examples below show two slightly different ways of filling out a journey map.  Matt’s team added the thoughts and emotions right next to the actions. Sophie’s team tried to stick to the vertical rows and separate these visually. These variations show that journey maps can be easily customized. 

From the application of the journey maps for the design of the UTM application the following design implications were identified and should be implemented in a second iteration:

(1) Workflows and input: Show an overview of the steps for multistep actions.

Issue:  “Where do I start my journey? How can I create the flight plan and then take off?” These were the questions participants oftentimes were wondering about during the test. This implies that the app did not show the steps to create a mental model for the process.

Solution: If we think about design patterns, an alternative solution could be to show the steps in a collapsed content sort of way: the steps are listed in their separated fields and if the user taps on the field, it takes him/her to a different page which is not actually a completely new one that would take extra time to load.

(2) Symbols, buttons: Use clear labels to tell users what to do next.

Issue: We had a button for Landing. It looked like a download button, hence people got confused. It was the only main button on the screen so they went for it, but they all recited that the button looked weird and out-of-place.

Solution: The participants recommended to spell out the actual letters “LAND” on the button, thus creating “call to action.

(3) How to save the checklist: Save/Submit button at the bottom supports normal flow.

Issue: Once the end users finished reviewing the pre-flight checklist, some of them were surprised that the “Save” button was in the upper right corner.

Solution: The users would have expected it at the bottom, so when they completed the list they could have just easily clicked on it.

(4) Navigation & Information Architecture.

Issue: The prototype had persistent bottom navigation, i.e. it did not move as users scrolled. There were however pages where it disappeared. This happened at one point when the user took off with the drone.

Solution: One user expressed the need to still be able to access the navigation bar for checking the weather or messages during the flight.

(5) Content: Make difficult content easy.

Issue: The critical information related to flight restriction was an in-app notification. The only problem was the way the text was worded. The author tried to convey a complex information with the most important part being at the end. But people rarely have the time to read carefully a whole paragraph on a mobile device.

Solution: It’s best to strive for short sentences that start with the essential.