Key Concepts

• Human HAZOP is a Group method, rather than a single-analyst approach such as TRACER or STAMP.

• It relies on ‘structured brainstorming’ by a small group (e.g. 6-8) design, operational and Human Factors experts, led by a HAZOP chairman

• It requires a representation of the system, procedure or interface being evaluated. Typically design documents and drawings (or photos if operational) are present, along with a task analysis 
   (typically Hierarchical Task Analysis or Tabular Task Analysis which details how the human operators are intended to interact with the system.

• The HAZOP study group proceeds through each step of the procedure or task analysis and considers potential deviations from the expected behaviour, prompted by HAZOP guidewords.

• The consequences of deviations from the intended functioning of the system are considered, as well as existing safeguards & recovery means

• Required safety recommendations are stated and recorded by the HAZOP secretary.

• HAZOPs are formally recorded and logged.

Benefits

HAZOP has survived a long time, and has a long list of benefits:

• Human HAZOP provides systematic and exhaustive design review, and can lead to the discovery of new hazards.

• It requires limited technical training – it is an ‘intuitive’ method.

• The use of a team gives a range of viewpoints.

• It has a good track record in several industries.

• It is versatile, and can be applied to all sorts of design formats.

• It can be used early on (e.g. at the concept design stage).

• It is good at finding credible vulnerabilities, including violations.

• It has high acceptance by both designers and operators, as they both input to the HAZOP.

• It creates a good ‘bridge’ between design and operations.

On the negative side, there are several well-known disadvantages:

• HAZOP, especially if applied to a complete system, is resource intensive.

• The GIGO (garbage in, garbage out) principle applies. If, for example, inexperienced operational people are used as experts, don’t expect too much.

• HAZOP tends to concentrate on single deviations or failures – it does not look at how different failures might interact – only risk modelling does this.

How It Works

Human HAZOP works by applying a set of guidewords to the chosen design representation, be it design drawings and documentation, task analyses, or more dynamic visualisations (e.g. working prototypes) of the system under investigation.

The guidewords are as follows shown in the table below. As an example, the most commonly used guideword is ‘No’, meaning the human operator does not do something when required (e.g. failing to lower landing gear). Guidewords such as More or Less could be applied to entering correct weight into software used to manage balance (aviation) or ballast (maritime). Reverse is rarer but can still happen, as in the AF447 air crash where the co-pilot manoeuvred the controls in the wrong direction, possibly due to startle response and inexperience, putting the plane into an irrecoverable stall. In maritime it could mean setting thrusters in the wrong direction when docking. Early and Late could relate to use of flaps or deciding when to descend an aircraft, or when to turn to pass another vessel (maritime). The most interesting guideword is ‘Other than’, wherein a completely unintended action is the result. This requires operational experience rather than fanciful imagination, and typically in a HAZOP only one or two (or none) such possibilities are identified. 

In the HAZOP session, the HAZOP Chairman leads the group through each step of the activity associated with the interface or procedure, and each guideword is considered in turn. 

The results of a HAZOP are logged in a HAZOP table by the HAZOP secretary. Mitigations may not always be apparent, in which case actions are given to HAZOP team members or their departments to develop mitigations and report back to the group or design authority to ensure they are sufficient and mitigate the hazard.

Illustrative Example

Aviation domain
The new Air Traffic Controller systems shift the focus away from one primary form of information presentation and usage (paper) to a completely different presentation medium (computer screen).  Additionally, although both these media are visual in nature, the computerisation of flight progress strips paves the way for automated ‘up-linking’ of messages to the cockpit of aircraft, reducing the need for oral communication for a number of tasks. This transmission of information is known as ‘data-link’. Given the high performance and proven nature of the current system, it is sensible to evaluate the transition to a new system interface.  A HAZOP analysis was therefore conducted on the proposed HMI (Human Machine Interface) of the new (electronic) flight progress information system. 

The HAZOP team that assessed the implications of the new interface was made up of three designers, one air traffic controller and two human factors specialists. The study identified a number of ‘vulnerabilities’ in the prototype system and ‘opportunities’ for error that needed to be designed out or worked around (e.g. via procedures and training).  A total of 87 recommendations, covering changes to the HMI, improved feedback and training / procedure improvements were generated in the three HAZOP sessions.  

The HAZOP approach proved surprisingly useful and productive of agreed changes in interface design. Moreover, since the designers were not only present, but were actively involved, any design changes they thought necessary were simultaneously accepted for implementation. The HAZOP therefore had very fast and effective impact on the design process. To illustrate the meeting output, part of the study is shown in the table below. 

Maritime domain

In shipping, the techniques to represent the geophysical characteristics of the seabed have shifted the focus away from paper-based nautical charts to computerized electronic charts such as sonar charts. A sonar chart is an HD bathymetry map featuring extraordinary bottom contour detail for marine and lakes, excellent for increasing awareness of shallow waters and for locating fishing areas at any depth level.

Although both techniques are visual in nature, by computerising the seabed mapping, it is possible to increase the visualization details and to reduce the need for communication and coordination amongst crew members. Therefore, a Human HAZOP analysis can be conducted on the proposed human-machine interface of the new electronic system. The outcomes of this study were used to identify a number of ‘vulnerabilities’ in the proposed system and ‘opportunities’ for error that needed to be designed out or worked around (e.g. via training in the use of sonar chart).To illustrate the HAZOP output, an example is shown in the table below.

Key Concepts

The TRACEr technique is based on the human information processing paradigm which claims that the human mind is like a computer or information processor. It contains eight interrelated taxonomies that altogether describe:

The context of the incident – using:
1. Task Error – describing controller errors in terms of the task that was not performed satisfactorily.
2. Information – describing the subject matter or topic of the error.
3. Performance Shaping Factors (PSF) – classifying factors that have influenced/ could influence the controller’s performance, aggravating the occurrence of errors or assisting error recovery.

The cognitive background of the production of an error – using:
4. External Error Modes (EEMs) – classifying the external and observable manifestation of the actual or potential error.
5. Internal Error Modes (IEMs) – linked specifically to the functions of the cognitive domains and describing which of the cognitive functions failed/ could fail and in what way.
6. Psychological Error Mechanisms (PEMs) – describing the psychological nature of the IEMs within each cognitive domain.
7. Error detection – describing the error using specific keywords.

The incident recovery – using:
8. Error correction – describing how the error was mitigated.

Benefits

• The TRACEr technique provides feedback on organisational performance before and after unwanted events. 
   Its main strengths are comprehensiveness, structure, acceptability of results and usability.
• It requires moderate resources.
• It offers a possibility to derive error reduction measures.
• It helps to determine what errors could occur, their causes and their relative likelihood of recovery.
• It facilitates the identification and classification of human errors in relation to Human-Machine Interaction (HMI).

How It Works

TRACEr can be used in two different ways, for a retrospective analysis (i.e. following an incident or accident) or for a predictive analysis, in order to anticipate the possible errors, in the context of a risk analysis. For the sake of brevity only the retrospective use is here illustrated, but in two different instances: first the original one, developed in the air traffic control domain, then a modified one adapted to the maritime domain.

Aviation Application
The process for applying TRACER retrospectively in the ATC domain can be represented as a flowchart, and identifying six different steps.

• Analyse incident into 'error events', identifying the task steps in which an error was produced. 
• Task Error Classification, classifying the error with the Task Error taxonomy 
• Internal Error Modes (IEMs) Classification, deciding which cognitive function failed.
• Psychological Error Mechanisms (PEMs) Classification, identifying the psychological cause.
• Performance Shaping Factors (PSFs) Classification, selecting any PSFs related to error under analysis.
• Error detection and Error correction, identifying errors/ corrective actions by answering to four questions.

Once the analyst has completed step number six, the next error should be analysed. If there are no more “error events” then the analysis is finished.

Maritime Application
The TRACEr methodology was adapted to the maritime sector by the World Maritime University and named TRACEr-Mar, it consists of nine coding steps or classification schemes that can be divided into three main groups, which describe the context of the incident, the operator context and the recovery from the incident. The overall process can be summarized with a flowchart and table below provides a brief summary of the nine TRACEr steps.

Illustrative Example

Aviation domain

To see how the technique can be implemented in practice we may consider the following ATCO-Pilot communication scenario which resulted in a loss of separation minima.

SCENARIO:  
A very busy day, young and unexperienced ATCO, tired because of lack of sleep. ATCO issuing clearance for aircraft A: “ABC123 descend FL240”; Pilot A readback: “ABC123 descend FL220”; ATCO issuing clearance for aircraft B: “XYZ789 climb FL230”; Pilot B readback: “XYZ789 climb FL230”.

STCA starts (Short Term Conflict Alert – ground-based safety net intended to assist the controller in preventing collision between aircraft). ATCO: “ABC123 avoiding action turn right.”

A representation of this scenario is illustrated in the figure below.

ANALYSIS focused on the first 'error event' (ATCO’s error): 

Step 1: Develop the sequence of events and identify error events.
. Controller fails to notice error
. Controller is late to respond to STCA
. Controller gives indication in wrong direction

Step 2: Task Error classification 
. Controller Pilot communication error

Step 3: IEM classification
. Readback error

Step 4: PEM classification
. Expectation bias

Step 5: PSF classification
. Traffic complexity

Step 6: Error detection and Error correction

Maritime domain 

SCENARIO: On December 2012, the dry cargo vessel Beaumont ran aground on Cabo Negro on the north Spanish coast while on passage from Coruna to Aviles. At the time of the grounding she was proceeding at full speed, and the Officer Of the Watch (OOW) was asleep. An inspection of the vessel’s internal compartment quickly established that, despite being driven hard aground on a rocky ledge, there was no breach of the hull. The MAIB investigation identified that the OOW had fallen asleep soon after sending his night lookout off the bridge. Available bridge resources that could have alerted the crew and/or awoken a sleeping OOW were not used resulting in Beaumont steaming at 11.5 knots with no one in control on the bridge for over an hour. 

The table below presents an example of applying the TRACEr-MAR taxonomy to the task error: Officer Of the Watch fell asleep on the bridge.

Key Concepts

• HEART method is based upon the principle that every time a task is performed there is a possibility of failure and that the probability of this is affected by one or more Error Producing Conditions (EPCs) (e.g. distraction, tiredness, etc.) to varying degrees.

• Within HEART, those factors which have a significant effect on performance are of greatest interest. These conditions can then be applied to a "best-case-scenario" estimate of the failure probability under ideal conditions to then obtain a final error chance.

• By forcing consideration of the EPCs potentially affecting a given procedure, HEART also has the indirect effect of providing a range of suggestions as to how the reliability may therefore be improved and hence minimising risk.

Benefits

On the positive side, HEART presents a long list of benefits:

• HEART is rapid and straightforward to use.
• HEART has a small demand for resource usage.  
• It has a good track record in several industries such as nuclear, healthcare, rail, maritime, offshore and aviation industries. 
• It can be used early on (i.e. at the development stage).
• Several validation studies have been conducted and concluded that this technique produce estimations that are equivalent to those obtained by applying more complicated methods.

On the negative side, HEART also presents several well-known disadvantages:

• The EPC data has never been fully released and it is therefore not possible to fully review the validity of it.
• HEART relies to a high extent on expert opinion, first in the point probabilities of human error, and also in the assessed proportion of EPC effect. The final Human Error Probabilities are therefore very sensitive.

How It Works

HEART is an Additive Factors Model (AFM) that proposes the application of a multiplicative impact on human reliability calculations. This technique works under the assumption that any estimated reliability of task performance may be modified according to identified EPCs.

The HEART methodology generally consists of two assessment process steps (a qualitative and a quantitative step).

• The qualitative method is started by classifying the generic task based on the accidents data report.  After finding the general task, it then breaks it down to smaller parts called Error‐producing conditions (EPCs). 

• Once this task description has been constructed, the second step of this methodology is the quantitative method, wherein Human Error Probability (HEP) is calculated , usually by consulting local experts. To obtain the HEP using the HEART methodology, first it is necessary to obtain the nominal human unreliability, which belongs to the generic task. When the EPCs are determined, the multiplication number of each EPC is obtained.

Illustrative Example

Aviation domain
This example is derived from an application of CARA (Controller Action Reliability Assessment), an adaptation of HEART specific for the Air Traffic Control domain). In this case CARA is used to asses human performance for anticipated landing clearances during low visibility.

During periods of low visibility, for aircraft landing at airports, a support system was being designed to enable controllers to be provided with information that runways were clear and available for aircraft to land on when they may not be visible from the tower. One key area around the runway in this process is the ‘Obstacle Free Zone’ (OFZ) for which consideration was being given as to the form of alert a controller would receive if the following hazard occurred “The aircraft which landed and is moving clear of the runway stops beyond the trigger line (which enables the next aircraft to start to approach the runway) but not clear of the OFZ with the landing aircraft less than 200 feet above threshold.”

Simple event trees were developed and focused on human error probabilities of two key ATC tasks:

1. Controller identifies the aircraft which has landed is stationary within the OFZ.
2. A warning is issued to the landing aircraft that the aircraft is stationary in the OFZ.

To populate the event tree, CARA values were calculated for these two tasks. The values were produced by two human factors experts and drew on human reliability analysis and event trees from previous hazard analysis work. For task 1, three potential designs were considered and values calculated separately:

1. Use of an audible and visual alert
2. Use of a visual alert only
3. Use of no alert, and relying on controller scanning patterns

Examples of calculations of the human reliability are presented below.

Issue warning to aircraft 2 (Aircraft approaching runway)
Task Description and Assumptions: having identified that aircraft one (the aircraft already landed) is stationary, the controller must identify and warn aircraft 2. The error under consideration is that the controller does not warn aircraft 2, having identified that AC 1 is stationary.

Results: Calculation  = 0.003 x [(11-1) x 0.2 + 1] = 0.003 x 3 = 0.009

The analysis provided the following quantified outcomes for the Hazard using the event trees for the three design scenarios:

• Use of an audible and visual alert - 9.4E-03
• Use of a visual alert only - 1.4E-01
• Use of no alert, and relying on controller scanning patterns 3.1E-01

The analysis identified that much better reliability would be achieved using option A, a visual and audible alert, as there is not a reliance on the controller to scan the display to identify the hazard. It was beyond the scope of this work to validate and take these data forward in to the system design.

Maritime domain
In the illustrative example, the case of ship collision during passage through the waterway (narrow canal) is applied. Human based factors are analysed to systematically predict human error for each task. The collision occurred between a tanker ship and a livestock ship during the narrow canal passage. The tanker ship which were steering with 11,7 knot speed to the course of 244° in Marmara Sea in the direction of Dardanelles, have contacted with her starboard bow to port stern quarter of the livestock ship within 20 nm range of Dardanelles at 06:25 LT on the date of 01.10.2013. Development of the event was as follows; the tanker ship overtook the bulk carrier ship when she was steering close to the middle of the separation in the direction of Dardanelles. It has been observed that livestock ship was steering in the starboard bow of bulk carrier ship in direction of Dardanelles and close to outmost line of the separation where she was also in the range of about 0,6 NM with the tanker ship. It also has been observed that another ship (container ship) was steering in the starboard bow of the tanker ship within the range of 1 mile. 

At the result of investigation of ECDIS records and statements of the officer on watch, it has been perceived that; After taking over bulk carrier ship with the range of 0,3 NM and heading towards to container ship, the tanker ship has contacted with livestock ship which were steering outmost line of the separation. Even though small course alteration to port side that was made by the tanker ship, contact couldn’t be prevented due to sudden and big course alteration to the port side had been made by the livestock ship. The root cause is attributed to the mental fatigue, inadequate manning on the bridge, inadequate standards, lack of communication, poor risk evaluation, delayed collision avoidance manoeuvring, limited time, lack of knowledge and inadequate checking. 

The VTS (Vessel traffic services) was informed about the collision. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer (OOW) was keeping navigational watch at the time of collision. The Master of ship was informed just before of the collision. Helmsman was called by the Master before collision. The hierarchical task analysis was performed to understand relative activities/tasks during ship strait passage. Then, HEART is applied for quantification of human errors. Following table shows details including tasks, responsible person, GTT, EPC, APOE and calculated HEP for the case of ship collision during the narrow canal passage.

Key Concepts

The key concepts of the techniques are:

• Task analysis: To define the activities to achieve the goal. 

• Common Performance Conditions (CPC): Human cognition and action context are determined in accordance with CPCs which provide a well-structured and comprehensive basis for characterizing the circumstances under which performance is expected to occur.

• Control modes: Scrambled, Opportunistic, Tactical, Strategic modes. The control modes are linked with different failure probability intervals representing human action failure probabilities.

• Cognitive activities: These are relevant for work in process control applications and also provide a pragmatic definition for each. Communicate, compare, diagnose, execute, monitor, observe, etc. are some of the cognitive activities. 

• Context Influence Index (CII): This value can be calculated by deducting the number of “reduced” CPCs (i.e. CPCs likely to have a negative impact on performance in the situation) from improved CPCs.

• Performance Influence Index (PII): Specify proper weighting factors for cognitive functions such as execution, planning, observation, and interpretation.

• Cognitive failure probability (CFP):  This is probability of failure for each cognitive failure type.

The main focus of CREAM technique is based on 2 elements:

• human performance - how a system behaves over time and 
• cognition activity description of cognition activity in the system/process.

Benefits

The technique presents a quantitative approach to systematically predict human error for designated tasks and ascertain the desired safety control level. It offers a clear and systematic approach to quantification. The quantification problem is, in fact, considerably simplified because CREAM is focused on the level of the situation or working conditions rather than on the level of individual actions. 

The technique is based on a fundamental distinction between competence and control. A classification scheme clearly separates genotypes (causes) and phenotypes (manifestations), and furthermore proposes a non-hierarchical organisation of categories linked by means of the sub-categories.

How It Works

The technique includes both basic and extended versions. 

The purpose of the basic method is to produce an overall assessment of the performance reliability that may be expected for a task. The basic one consists of the following steps:

1. Scenario development: This step provides a variety of scenarios including time availability, condition of the work place, the environment, stress, noise level, collaboration of the crew, etc. The CPCs are determined according to the scenario.

2. Task analysis of the process/work: The objective of this step is to identify the tasks and sub-tasks of the process according to the hierarchical task analysis (HTA).

3. Determine CPC: These are used to characterise the overall nature of the task. The combined CPC scores are needed to figure out human performance. There are nine CPCs defined in the CREAM: Adequacy of organisation, working condition, adequacy of the man-machine interface, availability of procedures/plans, number of simultaneous goals, available time, time of day, adequacy of training and experience, quality of crew collaboration. For each task, a CPC level is determined.

4. Determine control mode: The control mode is determined from the combined CPC score. The control mode corresponds to a region or interval of action failure probability.

The purpose of the extended version of CREAM is to produce specific action failure probabilities. The following steps are used:

1. Scenario development

2. Task analysis of the process/work

3. Determine CPC

4. Identify context influence index (CII): The CII is used to quantify CREAM, in particular CPCs, in order to simplify calculation. This value can be calculated by deducting the number of reduced CPCs from improved CPCs (Akyuz, 2015; He et al., 2008).

5. Determine performance influence index (PII): This step provides PII values which were generated to specify proper weighting factors for entire cognitive functions such as execution, planning, observation, and interpretation. Each CPC has a different PII value which means there are different weighing factors (He et al., 2008)

6. Calculate cognitive failure probability (CFP): The CFP specifies human failure probabilities for each cognitive failure type in order to calculate HEP value. Once the nominal cognitive failure probability (CFP0) has been nominated for each sub-task, the CFP value (HEP) can be calculated respectively (Akyuz, 2015). The following equation is used.

CFP=CFP0 x 100.26. CII     

Illustrative Example

Maritime domain
In the illustrative example, the case of ship collision during passage through a strait is handled. Human factors are analysed to systematically predict human error for each task. The extended version of CREAM technique is used.  

Step 1 Scenario development: 
The collision occurred between a tanker ship and a livestock ship during the Dardanelle Strait passage at sea. The collision occurred during overtaking. The VTS (Vessel traffic services) was informed about the collision since it occurred in the Dardanelle Strait passage. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer was keeping navigational watch at the time of collision. 

Step 2 Task analysis: 
Hierarchical task analysis was created covering potential activities/works during ship strait passage.   

Step 3 Determine CPC: 
Assessor determined appropriate CPCs for each task according to the scenario and crew condition.

Step 4 Identify context influence index (CII): 
Assessor determined CII values for each task

Step 5 Determine performance influence index (PII): 
Assessor determined PII values for each task. 

Step 6: Equation was used to calculate the human error probability (CFP) 
for each task.

Following table shows task analysis including responsible person, relevant cognitive activity, cognitive function and calculated CFP for the case of ship collision during the strait passage (He at al., 2008; Apostolakis et al., 1988).

Step 7: Identify significant human error: 
Analyse which of the sub-task of the ship strait passage need further details and if there is any control measure to mitigate human error. 

In the view of analysis, sub-task 2.2 (Proceed inside the separation line) has the highest cognitive error rate where OOW (Officer on Watch) and helmsman failed to observe monitoring ship position. Likewise, sub-task 3.3 (Increase ship manoeuvring ability by consuming diesel oil and steering engines) is another critical activity where human error increased. The Master failed to enhance the ability of ship manoeuvring by consuming fuel oil (instead of diesel oil) during strait passage.  

Key Concepts

The key concept of eye tracking is to be able to identify exactly what someone was looking at, when, and for how long. Eye tracking can be used in isolation, or in addition to other methods and techniques such as simulation or as part of a task analysis.

Benefits

• Eye tracking outputs can be particularly powerful when communicating results. This is especially true when working on multi-disciplinary projects. A diagram clearly showing how often someone looked at something or the scan path they followed in order to make a decision can paint a very clear picture to non-specialists. As already stated, eye tracking outputs can be used to give us an insight into peoples’ cognitive processes which they would otherwise be unable to communicate.

• Eye tracking outputs, specifically video outputs, can be used during interviews as an aid to the interviewee: “can you explain what you were doing then?” or “what was going through your mind during this action?” The prompt of seeing their actions again ‘through their own eyes’ can often trigger the interviewee to remember specific thoughts or feelings and allow a deeper insight into the actions at that time.

• The main downside of eye tracking is the requirement of specialist equipment, and the need acquire a specific product for analysing the data depending on the eye tracker product. This is a substantial outlay, but they are also available for hire via short-term licenses. The actual use of the eye tracking equipment also has some downsides. Depending on the product used, some of the head mounted glasses-type eye trackers can obstruct peripheral vision as the frames can be quite thick. Some newer models are frameless for this reason. Also, the equipment requires calibration before use, and can easily de-calibrate if the user touches the equipment or moves suddenly.

• Another point for consideration is the analysis of data. Eye tracking produces large quantities of data that quite often requires substantial ‘cleaning’ before they can be analysed. This takes time, but when this is done the actual analysis is fairly straightforward and most of the software packages are user friendly.

How It Works

Eye tracking requires specialist software to both gather and analyse data. The good news is that there are now some extremely good quality, affordable options, which has opened up this area of analysis, which in turn has led to more research in the area and an increased understanding of outputs and metrics.

Typically there are two types of eye tracking device: head-mounted, or screen / panel mounted. 

• Screen or panel mounted devices are typically used where the operator is in a fixed position and normally looking in one direction. These devices are used a lot for User Experience (UX) and web design applications to elicit where a user is looking in order to design webpages more effectively. These devices are also sometimes used in activities such as car driving, as the user is sat in one position and typically looking in one direction. 

• Head mounted eye trackers are worn by the operator and are a lot more versatile as the operator can move freely. These head mounted trackers can resemble a pair of glasses, and have multiple cameras tracking the operator’s pupils and eye movements, and also filming what they are looking at. Both types of device can be used in lab-based or applied settings since they are relatively non-intrusive. 

Gathering data is fairly straightforward and depending on the device, can produce many different metrics. The most common eye tracking metrics are based on fixations and / or saccades. Fixations are discrete stable points of where the eye is looking for at least 300ms (and therefore visual information is processed by the brain), and saccades are defined as the eye movements between these fixations (where visual information is not processed). Typically, the first stage of analysis would be to separate the area the operator is working in or looking at into ‘areas of interest’. These then allow you to focus your analysis. Fixation metrics can include number of fixations, fixation durations, total number of fixations in one area of interest, fixation density, and repeat fixations. Fixation metrics can tell us a lot about the operator’s engagement with certain tasks or information. For example, if an operator is focussing on an area for a long time, it could indicate a higher cognitive effort. If an operator is fixating on many different points, it could indicate that the information the operator requires is scattered around, and we should aim to work out why, and if the presentation of information could be improved.

Saccade metrics can provide us with information regarding decision-making, but typically saccades and fixations are used in combination to provide scan-path data. Scan-paths can give us insight into how people go about looking for information, and allow us to analyse how information is gathered and used in certain situations. Scan-paths can be compared between operators, and between novice and expert users to understand the differences between approaches and the impacts these may have. We may be able to identify certain scan-paths that are always straight, and quick, indicating that this task is efficient. Conversely, we may be able to identify scan-paths which are erratic which may help us to understand any aspects of the task or information provision which isn’t currently working as well as it should. These metrics can provide us with some powerful insights into how work is carried out and can greatly assist with redesign of interfaces, displays, and tasks.

Illustrative Example

Aviation Domain:
As part of a larger project, we wanted to understand how pilots behaved in a complex situation containing multiple incidents. This was carried out in a full motion simulator, and a complex scenario was played out which resulted in the co-pilot flying the plane, a number of go-arounds, a runway change, and low fuel throughout the scenario. From the other data (cockpit data, and voice recordings) and through expert analysis, it was clear there were pilots who dealt with the scenario more successfully than others. By using eye tracking outputs, we were able to develop theories as to why some pilots performed better than others, and attach metrics to the data, such as dwell times on specific instruments, or scan-paths which were repeated time and time again. We were then able to triangulate this with the cockpit transcripts and understand why certain things were looked at certain times, or indeed, allow us to explore later during interview why they weren’t.This information helped to inform more advanced cockpit instrumentation designs, as well as informing emergency training requirements. 

As a specific example of the insights afforded by Eye Movement Tracking (EMT), it was possible to see how the Captain and First Officer were cross-checking each other during the escalating emergency scenario, and how they supported each other. Typically, the FO, whose main fixation was the PFD (Primary Flight Display)  was able to ‘offload’ a degree of the situation awareness to the Captain so that he could focus on diagnosing the problems affecting their flight. EMT was also able to clearly pinpoint the moment when each crew realised they were low on fuel, and how they used various cockpit instruments to analyse an electrical failure. In some areas instruments such as the PFD were fully supporting the flight crew, whereas in others there was clearly room for improvement.

Maritime Domain:
In this study, navigator’s eye movement on the navigation bridge simulator was analysed during passage through the Istanbul Strait. A measurement device called “EMR-8” was used for eye movement tracking. In the application, three examinee groups were selected and they divided according to their onboard experience level. Each group has 4 examinees in real-time bridge simulator. Group 1 consisted of deck department students who had up to 2.5 months sea experience onboard ship as a cadet. Group 2 included junior officers who had already completed 12 months training on-board ship. Group 3 consisted of oceangoing masters who had wide experiences at sea. 

The scenario about passing Istanbul Strait was given to each examinee. The visual field was divided into three parts; inside, outside and others. The inside part has three components; instruments, indicators and an engine telegraph. The outside part has three components; the sea condition, navigational equipment and target ships. The scenario was recorded by eye movement tracking. In the light of findings, Group1 (beginners), gave minimum attention to navigation euqipments, target ships, regulations and sailing routes. Group 2, the examinees had intermediate characteristics as a navigation officer between professionals and beginners. They had enough knowledge and ability to use navigational equipments, target ships, regulations and routes. They had a longer duration on the bridge compared to Group 1. On the other hand, Group 3 had high level experience and knowledge. The examinees gave utmost attention to navigational equipments, sailing routes, target ships and regulations. Group 3 could stand on the bridge for the longest time without losing concentration. 

Key Concepts

The NEUROID is based on data-mining and machine-learning algorithms to make the HFs neurophysiological models fitting each operator and minimize inter-user variability, therefore obtaining reliable operator’s assessment. More than 88% of all general aviation accidents are attributed to human error. Therefore, NEUROID generates an innovative and systematic approach to quantify and objectively measure HFs by taking into account, at the same time, the behaviours, emotions, and the mental reactions of the operators themselves, and integrating them with the data related to accidents and incidents investigations. 

The general concept at the base of NEUROID is that brain, body, and operator’s experience are reciprocally coupled, and that accurate assessment of the operator’s can only be achieved through a well-defined combination of all the available data. Each biological activity is regulated by the human Nervous System, therefore variations of such biological activities correspond to internal reactions because of modification of external (environment) and internal (mental, motivations, emotions, etc.) factors. There are numerous types of neurophysiological measures, such as Electroencephalogram (EEG, related to brain activity), Electrocardiogram (ECG, hearth activity), Electrooculogram (EOG, ocular activity), Galvanic Skin Response (GSR, skin sweating), and so on. Such neurophysiological measures can be seen as the physical interface of the NEUROID technique that will enable to gather insights about all the aspects relating to HFs of the operator like Mental Workload, Stress, Vigilance, and Engagement. The key concept of NEUROID is the Human Performance Envelope (HPE) taxonomy: instead of considering one or two single human factors, the HPE investigates a set of interdependent factors, working alone or in combination, which allows to completely characterize the operator. In other words, these concepts are proposed as performance shaping factors, which can differentially and interactively affect successful completion of a task. The HPE theory explicitly declares that boundaries exist where performance can degrade in line with the theoretical underpinnings for the considered HFs.

Benefits

Traditional methods to catch information about the operators’ HFs are usually based on self-reports and interviews. However, it has been widely demonstrated how such kind of measures could suffer of poor resolution due to a high intra- and inter-user variability depending on the nature of the measure itself (i.e. subjective). In addition, the main limitation in using subjective and behavioural measures alone is due to the impossibility of quantifying ‘‘unconscious’’ phenomena and feelings underlying human behaviours, and most importantly it is not possible to measure such unconscious reactions experienced by operators while performing working activities. In fact, the execution of a task is generally interrupted to collect subjective evaluations. 

On the contrary, neurophysiological measures exhibit an unobtrusive and implicit way to determine the operator’s affective and cognitive mental states on the basis of mind-body relations. Therefore, the benefit and advantage of the NEUROID technique is to use neurophysiological measures and machine-learning algorithms to overcome such limitations, especially to (1) objectively assess the operator’s mental states while dealing with operative tasks; (2) identify the most critical and complex conditions and correlated them with accident and incident investigations; and (3) create a closed-loop between the systems and operators (i.e. Joint Human Machine Cognitive system) to continuously and non-invasively monitor the operators themselves.

Finally, modern wearable technology can help in overcome the invasiveness (e.g. many cables, uncomfortable) of the equipment necessary to collect the neurophysiological signals from the operators.

How It Works

• NEUROID consists in 3 main phases as reported in the figure. 

• The neurophysiological signals are gather from the operators by wearable technology while dealing with working tasks. 

• Afterward, the neurophysiological data are processed through a series mathematical steps towards the machine-learning phase, in which each HF is classified. 

• Finally, the considered HFs are combined to define the Operator’s HPE, and integrated with the operator’s behaviour, and data related to accidents and incidents (e.g. SHIELD database and HURID framework) for 

  a comprehensive operator’s assessment which can be employed for different applications, for example real-time mental states assessment, or triggering adaptive automations.

Illustrative Example

Aviation Domain
To see how the NEUROID technique can be implemented in practice we can consider the following scenario where an Air Traffic Controller (ATCO) is managing the air traffic and at the same time we are acquiring the ATCO’s EEG, ECG, and GSR neurophysiological signals. In the first phase of the scenario, the ATCO can rely on the support of high-automated systems to manage high traffic situation (High). Then, the traffic demand returns to a normal condition (Baseline), but at a certain moment the automations crash (Malfunction) and the ATCO will have to keep managing the traffic, and find out what is wrong with the automations.

In this context, we can use the NEUROID to measure how the ATCO’s HPE changes. In this regard, the neurophysiological data will be employed to estimate the ATCO’s Mental Workload, Stress, Attention, Vigilance, And Cognitive Control Behaviour based on the S-R-K model. Finally, those HFs will be combined to define the ATCO’s HPE on the three operational conditions as shown in the following figure. In particular, the values of the considered mental states have been normalized within the [0 ÷ 1] range: “0” means Low, while “1” mean High. Concerning the S-R-K aspect, “0” represent the S (Skill), “0.5” the R (Rule), and “1” the K (Knowledge) level.
 

The analysis of the HPE in the different scenario conditions exhibits interesting variations on the configuration of the mental states. In fact, the failure of the automations (red line) induced higher Vigilance, lower Attention, and shift from Skill to Rule level in the Cognitive Control Behaviour with respect to the HIGH (orange line) and BASELINE (blue line) conditions. Furthermore, considering the HPE changes over time throughout the ATM scenario allows to identify those moments, thus situations, corresponding to high (RICH) or low (POOR) performance. For example, for the considered ATCO, the HPE corresponding to RICH (green plot) and POOR (red plot) performance are reported in the following figure. Generally, in correspondence of RICH performance condition, the vigilance is higher and the stress is lower than during POOR performance. In conclusion, the NEUROID could be used, for example, to monitor the operators and warn them when they are working under low vigilance and high stress condition.

Maritime Domain
To see how the NEUROID technique can be implemented in Maritime practice we can consider the following scenario where the collision occurred between tanker ship and livestock ship during Dardanelle Strait passage at sea. The collision occurred during overtaking. The VTS (Vessel traffic services) was informed about the collision since it occurred during Dardanelle Strait passage. The weather was partly cloudy and sea state was calm at the time of collision. The time was early morning. The officer was keeping the navigational watch at the time of collision. It was early morning at the time of collision.

The HF analysis of the ship collision during Strait passage would highlight the following potential root causes: 

• Mental Fatigue: Rather than physical fatigue, over alertness and overload (decision making, concentration, mental load). Watchkeeper Officers continuously working load can affect his/her decisions.

• Physical Fatigue: Rather than mental fatigue, physical fatigue can be seen due to intensive work load on-board ship. 

• Inadequate Manning on Bridge: According to sailing zone, increase the number of watchmen on bridge may help perception of the dangers earlier.

• Inadequate Policy, Standards and Application: Increase of the inspections due to sailing rules and minimum CPA-TCPA values and activation of the existing rules may help avoiding dangerous situations and provide clearance from the plotted vessels.

• Inadequate / Lack of Communication: Necessary manoeuvre could not be applied due to inadequate communications between vessels. Lack of communications between vessels and VTS also caused vessels to sail in close range. Every communication methods on board must be managed in most efficient manner.

• Poor Risk Evaluation: Main reasons of the incident between vessels are the failure of the evaluation of risks for all vessels in the sailing area and unforeseen of the other vessel possible manoeuvres which were sailing in the outmost line of the separation. 

• Inadequate Leadership and Supervision / Planning: Another root causes of the incident are improper planning of watch hours and not taking necessary measurements as a result of evaluations of inconveniences timely.

• External/Environmental Conditions: Sailing in narrow zone, congestion and poorness of the VTS stations in planning the traffic are the main effective reasons of the incident. To prevent such like incidents, environmental conditions must be evaluated sufficiently and proper speed and watch order must be planned according to congestion.

Some of those aspects are mainly linked to the management level (e.g. manoeuvres procedures, risk evaluation), but the ones coming from the evaluation of the operators’ status like the mental and physical fatigue can be monitored and measured via the NEUROID during the working activities. 

In particular, along each phase of the considered scenario the neurophysiological measures can characterise the operators in terms of Mental Workload, Stress, Vigilance, and Engagement combination and those information can be consequently used to:

• interact and\or intervene in real-time on the system, for example depending on the Master's mental workload it could be possible to enable a warning during overload or low vigilance conditions to regain the proper operational status;

• post-doc analyse the incident\accident or simulation by combining the previous tasks analysis, and the operators' mental states profile along the scenario, for example to find out how the Mental Workload, Stress, Vigilance, and Engagement of the Master and C/O were right before the collision or during a specific phase which brought to the collision, and finally understand if the configuration of those mental states was not appropriate (e.g. distracted?) or, on the contrary, the operators were under too high demanding and stressing conditions due to the adopted procedure or situation.

Key Concepts

Safety culture has a number of dimensions, such as the following (Kirwan et al, 2016):

• Management commitment to safety (extent to which management prioritize safety);
• Communication (extent to which safety-related issues are communicated to staff, and people ‘speak up for safety’);
• Collaboration and involvement (group involvement and attitudes for safety management);
• Learning culture (extent to which staff are able to learn about safety-related incidents);
• Just culture and reporting (extent to which respondents feel they are treated in a just and fair manner when reporting incidents);
• Risk handling (how risk is handled in the organisation);
• Colleague commitment to safety (beliefs about the reliability of colleagues’ safety behaviour);
• Staff and equipment (extent to which the available staff and equipment are sufficient for the safe development of work);
• Procedures and training (extent to which the available procedures and training are sufficient for the safe development of work).

Safety culture is typically evaluated using an online questionnaire of validated questions that is distributed across the organisation and filled in anonymously (usually taking about 10 minutes). Based on the responses, analysts then identify the safety culture ‘hotspots’, which are then discussed in confidential workshops, leading to recommendations for the organisation to improve its safety culture. At a macro level, typically a ‘spider diagram’ is produced based on the results, highlighting which safety culture dimensions need most attention. In the example figure to the right, ‘Colleague commitment to safety’ is doing well in safety culture terms, whereas ‘Staffing and Equipment’ requires attention.

Benefits

Safety culture surveys, particularly when anonymous and carried out by an independent organisation, are often seen as one of the only ways to find out all your risks and what people really think about the state of safety and safety culture in an organisation. They can pinpoint both current issues that can be corrected (so-called ‘quick wins’) without too much trouble, as well as deeper problems that may be entrenched in the organisational culture. Most safety culture surveys do identify the safety culture strengths of an organisation as well as vulnerabilities, so there is always some ‘good news’ as well as areas for improvement. Because the approach uses workshops of operational personnel, usually the recommendations that arise from surveys are useful and practicable, since they come from the organisation itself. 

The survey process is long, however, often taking 9 months from the first decision to have a survey, to having a report with recommendations. The management need to be committed to doing the process and realise that it is not a quick fix, and there will be work to do afterwards. If management are doing the survey just to gain ‘a tick in the box’, then it may be better not to do one at all.

How It Works

In many application of the aviation domain, there are two main phases used to evaluate the safety culture in a safety culture survey:

Phase 1: Questionnaire 
The safety culture survey uses an electronic version of the Safety Culture Questionnaire, over a period of typically 3 weeks. The confidential data are anonymised by The London School of Economics and then independently analysed by EUROCONTROL

Phase 2: (Optional) Workshops 
After an initial analysis of the questionnaire results, there is a visit to the organisation’s premises to hold a number of confidential workshops with front-line staff (without managers present) to clarify and explore further certain aspects emerging from the questionnaires. An example of what might be discussed in such a workshop is indicated by the figure below, which shows results for an organisation in the safety culture dimension ‘Staff and Equipment’. With respect to the questionnaire statement ‘We have the equipment needed to do our work safely’, whilst more than 70% were in agreement (the green bar), the rest were either unsure or in disagreement. The workshop would explore equipment issues or concerns people have, which are underlying this response.

Illustrative Example

Aviation domain
It should be noted that the example reported was seen as an exploratory survey and were focused on a single particular segment of the organization – Airbus Design rather than being company-wide (see Kirwan et al, 2019).

For the Airbus survey, the focus was on the Design part of the organization, which involves around 2,000 of the 70,000 employees. This survey required considerable tailoring of the questionnaire, as the job of design and systems engineering is somewhat different from airline operations, even if most underlying issues, except fatigue, and dimensions remain relevant. But the tailoring worked, with the same issues raised during workshops as via the questionnaires. Participation in the workshops by designers from all three of Airbus’ main design locations (France, UK, and Germany) was intense. The general principle espoused in each workshop was that a safety defect in a design of an aircraft could simply not be allowed to happen; therefore, any safety issues had to be resolved carefully and thoroughly before continuing, even if resolution efforts delayed progress toward production. One observation from the study team was that the managers in the management workshop were connected with safety and detailed safety concerns, more so than is often the case for middle managers. This is because many of the managers had held operational positions or positions ‘close to operations,’, so that they understood the operational pressures that could affect airlines and pilots. Another had been to an accident site after an air crash, and noted that after such an experience ‘safety never leaves you.’

The workshops focused a lot on internal communications between the different design departments, as well as the resolution of safety issues that could delay production, and the importance of middle manager support during such resolution periods. 

A number of constructive recommendations were made to Airbus in the final survey report.
 

Maritime domain
A safety culture assessment and an implementation framework were developed by the University of Strathclyde to enhance maritime safety. The focus was on the development of a safety culture assessment tool which covers all of the safety related aspects in a shipping company and measures will be taken proactively and reactively in order to enhance safety of the shipping industry.

Questionnaires were developed with an interdisciplinary group to ensure that they capture the fight information to conduct a comprehensive analysis. Each safety factor has specific questions which try to address the employee’s attitudes and perceptions amongst crew members and shore staff. There are a total of 85 questions which are asked to the employees, under the headings of communication, employer-employee trust, feedback, involvement, problem identification, promotion of safety, responsiveness, safety awareness and training and competence. 

The aim of the surveys is to analyse strengths and weaknesses in the company and perform benchmarking between crew and shore staff’s attitudes towards safety. Participants of the surveys were asked questions about demography, safety factors and open-ended questions about company policies. The safety culture assessment framework requires commitment from all of the bodies in a shipping company for successful implementation. 

Based on the findings from the questionnaires a set of recommendations were provided to the shipping company to improve communication, employer-employee trust, feedback, involvement, mutual trust, problem identification, responsiveness, safety awareness, training and competence, and the promotion of safety.

Key Concepts

The LOAT brings into discussion important matters related to systems’ innovation.

Automation is not only a matter of either automating a task entirely or not but deciding on the extent to which it should be automated.

Introducing automation brings qualitative shifts in the way people practice rather than mere substitutions of pre-existing human tasks.

Classification of different levels of automation, recognises various ways in which human performance can be supported by automation.

The “optimal” level of automation in a specific task context is about matching the automation capabilities to a number of operational situations, while increasing the overall performance in efficient human-machine cooperation.

It provides potential alternatives for designing the human-machine interaction by taking full benefit of the technical solution.

Benefits

The LOAT can be used to:

Compare different design options and determine the optimal automation level in all operational contexts. The best level of automation depends on the capabilities of the automation itself and on the operational context in which it is included. You can use the LOAT to orient yourself in these choices.

Understand the impact of automation, both positive and negative, on the system. The highest level of automation is not always the best one. Finding the right level helps to make sure that you are taking the full benefit from it, without being negatively affected by side effects, such as nuisance alerts or erroneous directions.

Get insight on how to prevent human performance issues and take full benefit of available technical solutions in various domains. A common taxonomy for different domains will mean a shared approach to automation design. Developed in the Air Traffic Management domain but applicable to all domains in which automation plays an important role in sustaining human performance.

Classify existing implementations and derive lessons to improve automation design. Classifies the ‘nuances’ of automation enlightening the possibilities in design. Modifying existing tasks or introducing new ones involve the use of different psychomotor and cognitive functions, which implies the adoption of different automation solutions.

How It Works

The LOAT is organized as a matrix:

I. in horizontal direction: the 4 generic functions, derived from a four-stage model of the way humans process information:
1) Information acquisition: Acquisition and registration of multiple sources of information. Initial pre-processing of data prior to full perception and selective attention;
2) Information analysis: Conscious perception, manipulation of information in working memory. Cognitive operations including rehearsal, integration, inference;
3) Decision and action selection: Selection among decision alternatives, based on previous information analysis. Deciding on a particular (‘optimal’) option or strategy;
4) Action implementation: Implementation of a response or action consistent with the decision previously made. Carrying out the chosen option.

II. in vertical direction: each cognitive function groups several automation levels (between 5 and 8). All automation levels start with a default level ‘0’ - corresponding to manual task accomplishment - and increase to full automation.
 

The initial methodology that has been defined for the LOAT incorporates 4 steps: 1) Identify the automated tool to be classified; 2) Determine which function is supported (can be more than one); 3) Determine the relevant cluster of automation levels; 4) Identify the specific automation level and consider the design principles associated to it. The main LOAT goals are:

• it is a method by which we can efficiently identify the most suitable human-automation design;
• based on the classification of already developed automation examples we can provide specific and relevant human factor recommendations.

Illustrative Example

Aviation Domain
To see how the LOAT can be implemented in practice we should consider an example of human-machine interaction in industry.

SCENARIO: The Remote (or virtual) Tower uses an array of sensing and communication technologies on the airfield and the surrounding airspace, while providing the actual control functions (using certified controllers). In matter of automation the remote towers come with the following technology:

• surveillance is generally provided by an array of video cameras, both conventional and infrared, to replace and improve upon the controller’s OTW view.

• communications between controllers at the RTC and aircraft that use the airport are primarily via radio.

• displays typically include an “out-the window” display of the airfield, data from radar (if available) and any other surveillance method. They may also include equipment indicating weather conditions (such as a wind rose), a compass rose, and geographical overlays.

• controls would include microphones lighting controls, camera controls, etc.

• other technologies include object tracking and alerting (in the air and on the ground) and features such as automatic runway intrusion alerts.

As in a conventional tower, all surveillance and communications information is transmitted to one or more controller positions in the Remote Tower Center (RTC).

ANALYSIS: Following the aforementioned steps, we can use the LOAT to identify the level of automation.

• Step 1: Identify the automated tool to be classified. From the technologies of the RTC mentioned in the SCENARIO we choose the one we want to classify: e.g. Surveillance

• Step 2: Determine which function is supported. We should then consider the main purposes of surveillance cameras: i.e. they collect images. Considering this, the function supported is: Information Acquisition (A).

• Step 3: Determine the relevant cluster of automation levels. While the cameras are the ones collecting and generating the image to the user but only on the user’s demand, the relevant automation level is: supported by Automation. 

• Step 4: Identify the specific automation level. The user is still in control to decide whether the information received from the cameras is relevant or not, and can filter the information received, so the automation level is: with user filtering and highlighting of relevant info (A2) 

In this way the LOAT can be used to decide upon the optimal level of automation suitable for the user and can help to establish whether or not it is a good choice to automate the system in the first place.