RO-2017-102 Signalling irregularity, Wellington Railway Station, 3 April 2017

1. During the morning peak period on 3 April 2017, trains in the Wellington Railway Station yard were being controlled by two signal box operators (signallers) from the Wellington signal box.
2. Signal maintainers were at the time working on a track signalling fault in the area.
3. The signallers were setting the route for a shunt locomotive to recover a set of empty passenger carriages from Platform 9. The signalling fault prompted the signallers to use an alternative route for the shunt locomotive to access Platform 9.
4. However, the route they chose was not physically possible due to the design configuration for some of the crossing points in the area. The signallers were not aware that the route they had set was not possible because the mimic screen they were referring to in the signal box erroneously displayed their chosen route as valid (possible).
5. As a result of the way the points were set, the shunt locomotive was diverted down another section of track into a potential head-on collision with a departing passenger train. The driver of the shunt locomotive realised that his train was routed down the wrong track and stopped his train, but not before it encroached onto the track where the loaded passenger train was heading. Both trains were approaching a red signal that would have required them to stop anyway, but potentially only metres apart.
6. The driver of the passenger train saw the red light, and a train encroaching onto his track ahead, and stopped his train. There was no collision and nobody was injured
7. The Transport Accident Investigation Commission (Commission) found that the error on the mimic screen was the result of an equipment upgrade programme that had overlooked a design aspect of one set of crossing points and had not ensured that the new system was fully representative of the actual track layout. The error with the mimic screen and a missing interlock had gone undetected for many years.
8. The Commission also found that neither of the signallers on duty at the time was aware that they had wrong-routed the shunt locomotive, because they were unaware of the limitations of the crossing points that prevented the shunt locomotive travelling along the chosen route.
9. The Commission also found that the signal box did not have an interlock to prevent the signallers setting a route for the shunt locomotive that was not physically possible due to the design of one of the crossing points. An interlock would have prevented the incident.
10. The Commission identified two key safety issues:

• the change management process for upgrading the signal box display had not ensured that the mimic screen matched the physical track layout, and had not detected the absence of an interlock to prevent the signallers setting the points to a configuration over which it was not possible for rail traffic to travel
• the signallers lacked some familiarity with the physical layout and equipment capabilities in the Wellington yard that they were controlling.

1. The Commission made two recommendations to KiwiRail Limited to address these safety issues.
2. Key lessons arising from this inquiry were:

• when changes are made to safety-critical systems, the new systems should be fully tested for correct functionality, and the users of the systems should be trained in and familiar with them
• personnel controlling the movement of rail traffic should follow procedures rather than make assumptions regarding the status of signalling equipment.

1. On Monday 3 April 2017 the NZ Transport Agency notified the Transport Accident Investigation Commission (Commission) of the incident. The Commission opened an inquiry under section 13(1)b of the Transport Accident Investigation Commission Act 1990 to determine the circumstances and causes of the occurrence, and appointed an investigator in charge.
2. Commission investigators conducted a site examination on 3 April 2017, including an examination of the Wellington signal box to obtain photographic evidence.
3. Investigators interviewed the train driver on Tuesday 4 April 2017, the remote control operator (RCO) (the operator of a shunt locomotive. The locomotive is controlled by the operator, who stands in a protected area at the front of the locomotive using a remote control unit) on Wednesday 5 April 2017, and the two signallers on Friday 7 April 2017.
4. The Commission obtained the following documents and records for analysis:
   • the signalling and interlocking diagram
     • the signal log output data for the movements of the shunt locomotive and the passenger train
     • witness statements and interviews
     • the training records for the signal box operators
     • the rosters and timesheets for the signal box operators
     • the record of operating incidents for the persons involved
     • the record of track and signalling faults in the Wellington signal box area from 2012 to the date of the occurrence
     • the commissioning records for the signal box mimic screen (an electronic display showing the status of signalling equipment and the locations of trains within a specific area) display installation in 2010
     • records of interviews with personnel involved in the design and commissioning of the mimic screen.
5. On 9 April 2018 the Commission approved the draft report for distribution to interested persons for comment.
6. Two submissions were received. The Commission considered the submissions, and changes as a result of those submissions have been included in the final report.
7. On 23 May 2018 the Commission approved the final report for publication.

Background

1. The incident involved a shunt locomotive travelling southbound from the mechanical repair depot to recover passenger carriages from Platform 9 at Wellington Railway Station. The intended route of the shunt locomotive is shown in red in Figure 1. The actual route taken by the shunt locomotive is depicted by the blue line.
2. At around the same time, a passenger train was scheduled to depart from Platform 6 and travel to Upper Hutt. The intended route of the passenger train is shown in black in Figure 1.
3. The train movements were controlled by signal box operators (signallers) from the Wellington signal box. The signallers used a system of control levers to change mechanically the positions of points and signals within Wellington Railway Station limits in order to set paths for trains.
4. The Wellington signal box display had been upgraded in 2010. The upgrade had included the installation of a visual display panel, known as a mimic screen, that provided information to the signallers on the status of signalling equipment. Prior to the upgrade the system had provided basic information regarding what section of track was occupied by a train, but it had not indicated any route setting for its movement. The mimic screen was the signaller’s primary tool for monitoring the status of the signals and points that were controlled from the signal box.

Narrative

1. On 3 April 2017 the Wellington signal box was being operated by two signallers during the morning peak period. Signaller A was the signaller in charge.
2. At about 0750 one of the signallers observed a track fault that affected train movements to and from two of the nine passenger platforms. The fault was reported for attendance by a signals maintainer (the maintainer), who arrived on site at about 0900.
3. Owing to the nature of the track fault, signaller B left the signal box to operate manually the points where the fault had occurred until the maintainer was on site. Signaller B returned to the signal box after the maintainer arrived.
4. On returning to the signal box, signaller B was tasked by signaller A to operate signal levers under his direction. Signaller A was involved in telephone and radio communications with train drivers and maintenance staff.
5. At 0932 a ‘proceed’ signal was given to the passenger train in readiness for its departure from its platform. However, the next signal remained at stop because there were other movements taking place at the time, which the signalling system recognised as a potential conflict.
6. At about 0935 the shunt locomotive was required to travel southbound to collect passenger cars from Platform 9. An RCO was driving the shunt locomotive from a safe riding position at the front of the locomotive, using a remote control unit. The RCO requested permission from signaller A to begin the move towards the platform.
7. Before the movement could commence, signaller A needed to set a route for the shunt locomotive by altering points towards the platform.
8. At about the same time, the Upper Hutt-bound passenger train was scheduled to depart northbound from Platform 6.
9. Signaller B was instructed by signaller A to set the route for the shunt locomotive. The selected route was that depicted by the red dotted line in Figure 1. The mimic screen showed that the route was set as signaller B had intended. However, the configuration of one particular set of points through which the shunt locomotive had to travel was known as a ‘single slip switch’ (a mechanical installation that enables rail traffic to be guided from one track to another from a single adjacent track (see Appendix 1 for further description).) (see Appendix 1). This configuration did not provide for rail traffic to follow the route displayed on the mimic screen. Unbeknown to the signallers, the route was set for the shunt locomotive to follow the path of the blue line, contrary to the mimic display.
10. The maintainer working at the location of the track fault confirmed with signaller A by radio that the points in his area were in the correct position for the shunt locomotive to proceed to platform 9. The maintainer then moved to a safe place clear of the track.
11. Signaller A glanced at the mimic screen inside the signal box and noted the route set for the shunt locomotive.
12. The shunt locomotive was stopped at a red signal. Signaller A gave verbal authority to the RCO on the shunt locomotive to pass the red signal and proceed to the next red signal near the track fault shown by the red dotted line in Figure 1. He gave verbal authority in the belief that the signalling system would not be able to provide a proceed signal because of the track fault.
13. At 0935:40 the RCO stopped the shunt locomotive when he realised that the track he was on was leading him to a different location from the one that had been authorised by signaller A (see Figure 3). He then informed signaller A by radio that the route had been incorrectly set.

1. At 0936:32 the Upper Hutt passenger train began moving away from Platform 6 under a proceed signal. Eleven seconds later the driver stopped his train as he observed both a stop aspect on the next signal and the shunt locomotive encroaching onto his track ahead (see Figure 4).

1. At 0938 signaller A instructed the driver of the passenger train to change ends and return his train to the platform.
2. At 0945 signaller B informed the network control manager (the network control manager, among other duties, is the supervisor of signal box staff. They are based at the National Train Control Centre at Wellington Railway Station) of the incident. The network control manager went to the signal box a short time later to investigate.

Key personnel

1. This incident was initially interpreted by the operator as a wrong-side failure (a failure within railway signalling equipment that results in an unsafe state). In this context it was suspected that there had been an equipment failure that sent the shunt locomotive to the wrong location. Consequently none of the staff involved underwent a post-incident drug and alcohol test.

Signaller A

1. Signaller A had been operating the Wellington signal box since March 2004.
2. On the day of the incident it was his first shift back at work after two rostered days off. He had started work at 0530 and at the time of the incident had been on duty for four hours.

Signaller B

3. Signaller B had commenced service as a trainee signal box operator on 11 July 2016 and gained certification on 18 October 2016.

4. Signaller B had started work at 0530 and been on duty for four hours at the time of the incident. The day prior to the incident signaller B had worked a nine-hour shift that commenced at 0450.

The passenger train driver

5. The driver of the Upper Hutt-bound passenger train had 28 years’ rail experience, including five years as a signaller. It was his first shift after two rostered days off.

The shunt locomotive driver (RCO)

6. The RCO had 15 years’ rail experience. It was his first shift after two rostered days off.

Introduction

1. The circumstances and causes of this incident can be described as a combination of active and latent failures leading to a near miss.
2. ‘Active failure’ is a term used to describe unsafe acts that can be directly linked to an accident, typically the actions of the people involved. ‘Latent failures’ are contributing factors that lie dormant in the wider system for days, weeks or, as with this incident, even years.
3. The potential collision was averted through the actions taken by both train drivers. The signalling system ensured that the signals in front of each train remained at stop.
4. The following analysis discusses the circumstances that led to the shunt locomotive being diverted into the path of the northbound passenger train. The analysis also discusses two key safety issues:

• the process for upgrading the signal box display had not ensured that the mimic screen matched the physical track layout, and had not detected the absence of an interlock (controls fitted between points and signals that prevent the signaller setting conflicting routes) to prevent the signallers setting the points to a configuration over which it was not possible for rail traffic to travel
• the signallers lacked some familiarity with the physical layout and equipment capabilities in the Wellington yard that they were controlling.

1. The incident occurred in an area where train speed was limited to a maximum of 20 kilometres per hour. The shunt locomotive was inadvertently routed towards a track that had been prepared for a departing passenger train.

What happened

1. The signalling system for the Wellington yard is a logic-based system that interprets electronic feedback from signalling equipment to determine if the correct conditions are met before actions are allowed. For example, if the system recognises that a set of points is in the correct position and the track in advance is unoccupied, a proceed signal will be given into that section when requested. Conversely, if the signalling system recognises that the required conditions are not met, a proceed signal will be rejected and the signal will remain at stop (red).
2. The signalling system can also be enhanced by interlocks, which prevent operators setting conflicting signalling conditions. For example, an interlock would not allow an operator to move a set of points if the signal controlling entry to that section were displaying green. In order to move the points, the operator would have to satisfy the safety conditions of the interlocking design by placing the signal to stop first.
3. Signallers are authorised to give verbal authority to train drivers to pass signals at stop if the signalling system is unable to provide a proceed signal. Because this is an override of the signalling system, procedures are in place to regulate its use (see section 4.4.3).
4. At the time of the incident there was a track fault, referred to as a ‘dropped track’ (a dropped track occurs when the signalling system shows an unoccupied section of track as occupied, usually due to an electrical circuit fault but potentially because of a broken rail. This is a method of fail-safe protection that prevents signals being cleared into the affected section or points being remotely moved from their current positions), that was affecting the normal operating method of directing trains into and out of platforms 8 and 9 (see Figure 5). The fault was causing the signalling system to read the track as being occupied and could not provide a proceed signal into the affected track section, or set the points for trains to enter the section. That was why the signaller or the maintainer was required to hand-wind the affected points rather than operate them from the signal box.

1. An alternative route existed that would have placed the shunt locomotive at the intended location. However, the alternative route utilised the main passenger running lines, and for this reason the signallers were in the habit of avoiding it if possible (see Figure 6). If this route had been set the signalling system should have been able to provide a proceed indication on the first signal facing the shunt locomotive, thereby negating the need for signaller A to give verbal authority to the RCO for the shunt locomotive to proceed past a red signal.

1. Instead, the route shown in Figure 7 was selected. However, because of the single slip switch design it was not physically possible for trains to travel over this route and there was no interlock fitted to the levers in the signal box to prevent it being selected. Furthermore, an inaccuracy in the mimic display allowed a valid route to be displayed. These two failures are discussed further in the following sections.

1. The signalling system also prevented the passenger train receiving a proceed signal because of the conflicting way the points had been set for the shunt locomotive. The signalling system performed as designed and therefore should have prevented a collision.

Latent failures in the signalling system

Safety issue – The process for upgrading the signal box display had not ensured that the mimic screen matched the physical track layout, and had not detected the absence of an interlock to prevent the signallers setting the points to a configuration over which it was not possible for rail traffic to travel.

Mimic screen

1. From their signal box, signallers were not able to physically see the entire Wellington yard. Therefore the mimic display was their primary tool for knowing the status of the signals and the routes that were set for trains. Any fault in the mimic display was therefore serious.
2. Figure 8 shows a screenshot of the mimic display in the Wellington signal box as it was observed by the signallers at the time of the incident. Based on the information displayed, they were led to believe that the shunt locomotive would follow the path of the red arrow, when really the route was set for the shunt locomotive to follow the blue arrow, towards the path set for the passenger train.

1. The display error had been a latent failure lying dormant in the system for seven years, and as far as could be determined it had never been noticed or reported.
2. The mimic screen had been designed in 2010 by an independent contractor. The process of commissioning the mimic screen had involved testing that electronic inputs from signalling equipment matched what was being displayed on the screen. The testing regime had included an electrical wire count and continuity checks, circuit checks and functional tests. All testing had proved correct in that the mimic screen was displaying exactly the information it had been provided. However, the layout of the physical track, specifically the location of the single slip switch, meant that an error had been introduced that was not identified until this incident occurred. The final acceptance of the system had been signed off by KiwiRail’s predecessor, New Zealand Railways Corporation.
3. The investigation found that the particular points configuration resulting in the mimic screen showing an impossible route through the single slip switch had not formed part of the commissioning process. No evidence was found to suggest that the single slip switch configuration had been considered at any time during the design, implementation and testing of the mimic screen.

1. Figure 9 is a schematic drawing of the signal box display panel before it was replaced by the electronic mimic screen display. This provided a more accurate display of the limitations of the single slip switch. The schematic shows that using number 42 points (shown in red) it is possible to cross the yellow line and continue through number 43 points (shown in blue) (the route the shunt locomotive took), but it is not possible to veer left onto the yellow line (where the signallers intended the shunt to go). However, the mimic screen gave the signallers the false impression that rail traffic could move from the red to the yellow lines as shown in Figure 8. A comparison between the schematic drawing and the functionality of the mimic screen would have revealed this anomaly.
2. The omission of the single slip switch configuration during the design, testing and commissioning processes was an oversight that had introduced a latent failure to the Wellington signal box system.

Track interlocking

1. There was one other set of points of the same single slip switch configuration in the Wellington signal box controlled area. This set of points was fitted with locking protection (an interlock) on the levers in the signal box to prevent any incorrect alignment. No such interlock had been fitted to the points involved in this occurrence. KiwiRail advised that it had checked all available records dating back to the 1950s and had been unable to establish why locking protection had not been fitted to both sets of single slip switches.
2. An interlock would have prevented the points being incorrectly aligned and consequently the mimic screen would not have displayed an incorrect route. The signaller authorised the shunt locomotive movement based on the incorrect information displayed on the mimic screen.
3. Having an interlock preventing the points being incorrectly configured was another important defence that would have prevented the incident – this was a latent failure that had gone unnoticed or undetected for at least 60 years. If the process for upgrading the signal box display had included a failure mode effect analysis, this omission may have been identified.
4. KiwiRail has since changed the mimic display so that it accurately reflects the functionality of the equipment in the yard, and fitted an interlock to the levers in the signal box to prevent signallers attempting to set incorrect routes through the single slip switch.

Active failures

Safety issue – The signallers lacked some familiarity with the physical layout and equipment capabilities in the Wellington yard that they were controlling.

1. Before even reaching the first set of points, the shunt locomotive had to pass a signal that was showing red. Signaller A said he had assumed that the signal would not be able to be placed at proceed because of the track fault in the adjacent section of track. Based on this belief, he did not instruct signaller B to change the signal to proceed by activating the request lever in the signal box. This was an incorrect assumption. The track fault was not affecting the performance of the signal and under normal circumstances, had the request lever been activated, the shunt locomotive would have received a proceed signal.
2. However, the signalling system was preventing the signal being placed at proceed because the route displayed on the mimic screen was not physically possible due to the design of the points. The route was partially set in the wrong direction, which the signalling system recognised as a conflict. The signalling system was working as designed. Having assumed that he would not succeed in obtaining a proceed indication on the signal facing the shunt locomotive, signaller A gave verbal authority for the shunt locomotive to pass it at red. KiwiRail had a procedure that the signaller was required to follow before doing that.
3. KiwiRail’s Operating Rules stated in part:

1. Although the rules did not stipulate the measures that signallers should take to ensure that all points were correctly set and secure for the intended movement, arguably the signallers in this incident complied with this requirement (the first bullet point). The maintainer had confirmed with signaller A that the points affected by the track fault had been manually set for platform 9, and the mimic screen in front of them was indicating that the route was correctly set, even though it was not.
2. The second to fifth bullet points were complied with; however, the last bullet point was not. The signal lever was not operated to clear the signal and left in that position. As mentioned above, the signallers did not do this because they were working on the assumption that they would not be able to clear the signal because of the track fault. Under the circumstances, this failure to follow procedure was not a factor contributing to the incident. Even if the lever request had been operated, the signal facing the shunt locomotive would have remained red because of the incorrect route setting. Not being able to clear the signal would likely have confirmed signaller A’s belief that he would be unable to clear the signal because of the track fault. Nevertheless, it is of concern that he did not follow that part of the procedure.
3. Notwithstanding the fault in the mimic screen, the signallers had a surprising lack of familiarity with the physical layout of the yard they were controlling, particularly as one of the signallers had some 13 years’ experience in the Wellington signal box. The fact that no-one was aware that the mimic screen was not representative of the actual yard layout is an issue. If anyone had been aware of the anomaly, it would be equally surprising that they had not raised the issue. Signallers’ familiarity with the Wellington yard layout is an issue that KiwiRail needs to address.

Summary

1. A number of events and conditions aligned on the day to allow the potential collision to occur. Fortunately an actual collision was averted due to the diligence of the two train drivers involved. However, to manage risk well and prevent similar incidents or actual collisions in future, it is better to address the safety issues further up the causal chain, the system issues, rather than rely on the final defence to prevent accidents occurring.
2. The lack of an interlock on the relevant control box levers and the errors that were made when the signal box was modernised to include a mimic screen to replace a schematic diagram were latent failures, and the absence of either could have prevented the wrong-routing of the shunt locomotive.
3. The presence of a track fault was a condition on the day that, in the busy peak period for the Wellington yard, created the need for the signallers to adapt the normal route for the shunt locomotive. Their lack of familiarity with the equipment in the yard was a knowledge-based issue, because they were unaware that the route they set for the shunt locomotive was not physically possible.
4. The signallers taking actions based on an erroneous assumption that the starting signal would be affected by the track fault was not a factor contributing to the wrong-routing. However, based on that assumption they omitted to follow a required procedure that in different circumstances could contribute to incidents and accidents in future.

1. The mimic screen in the signal box was the primary tool for the signal box operators to identify the location and status of rail traffic and signalling equipment, yet it was giving them erroneous information about the validity of the chosen route for the shunt locomotive.
2. Neither of the signal box operators on duty at the time was aware that they had wrong-routed the shunt locomotive, because they were unaware of the limitations of the single slip switch that prevented the shunt locomotive travelling along the chosen route.
3. The error on the mimic screen was the result of an equipment upgrade programme that had overlooked the single slip switch configuration and had not ensured that the new system was fully representative of the actual track layout.
4. The signal box was not fitted with an interlock, which would have prevented the signal box operators setting an incorrect route for the shunt locomotive.
5. The signalling system that recognised a potential conflict and kept the signals at stop was a further and final defence that prevented the trains colliding.
6. The vigilance of the drivers of the passenger train and the shunt locomotive resulted in both trains stopping before a collision occurred.

1. The Commission identified two key safety issues:

• the process for upgrading the signal box display had not ensured that the mimic screen matched the physical track layout, and had not detected the absence of an interlock to prevent the signallers setting the points to a configuration over which it was not possible for rail traffic to travel
• the signallers lacked some familiarity with the physical layout and equipment capabilities in the Wellington yard that they were controlling.

General

1. The Commission classifies safety actions by two types:

   (a) safety actions taken by the regulator or an operator to address safety issues identified by the Commission during an inquiry that would otherwise result in the Commission issuing a recommendation

   (b) safety actions taken by the regulator or an operator to address other safety issues that would not normally result in the Commission issuing a recommendation.

2. Safety actions addressing safety issues identified during an inquiry

   The signalling system

3. KiwiRail corrected the mimic display within 24 hours of the event.
4. KiwiRail advised the Commission on 8 January 2018 that a method of interlocking points levers to prevent a reoccurrence had been installed in the Wellington signal box.

Signal box personnel

2. KiwiRail advised that since 1 May 2017 enhanced auditing procedures had been put in place for signal box personnel consistent with those for train controllers.

General

1. The Commission may issue, or give notice of, recommendations to any person or organisation that it considers the most appropriate to address the identified safety issues, depending on whether these safety issues are applicable to a single operator only or to the wider transport sector. In this case, recommendations have been issued to KiwiRail.
2. In the interests of transport safety it is important that these recommendations are implemented without delay to help prevent similar accidents or incidents occurring in the future.

3. KiwiRail’s change management process for upgrading the signal box display had not ensured that the mimic screen matched the physical track layout, and had not detected the absence of an interlock to prevent the signallers setting the points to a configuration over which it was not possible for rail traffic to travel.

   On 23 May 2018 the Commission recommended that the Chief Executive of KiwiRail review KiwiRail’s change management processes for modifying existing and building new safety-critical systems, and ensure that these change management processes include a full failure mode effect analysis and require functional testing before the new or modified systems are put into service. (010/18).

   On 7 June 2018, KiwiRail replied:

   In response to recommendation 010/18 KiwiRail will add the following steps to the process for scoping, design and testing of significant changes to safety significant control systems:

• At the scoping phase the identification of potential failure modes in consultation with stakeholders via a safety in design risk assessment;
• Formal documentation of issues discovered and resulting changes undertaken throughthe testing process – this testing includes both technical and end-user;
• Formal technical and user signoff before changes are released for implementation; and

• A post implementation review and associated issues tracking to capture and resolve and residual issues.

  Functional testing of changes is already part of existing processes.

1. Notwithstanding the fault in the mimic screen, the signallers had a surprising lack of familiarity with the physical layout of the yard they were controlling, particularly as one of the signallers had some 13 years’ experience in the Wellington signal box. The fact that no-one was aware that the mimic screen was not representative of the actual yard layout is an issue.

   On 23 May 2018 the Commission recommended that the Chief Executive of KiwiRail review KiwiRail’s system for training and ongoing performance monitoring for signal box operators to ensure that they are fully familiar with the capabilities of the equipment and the layout of the yards they are controlling. (011/18)

   On 7 June 2018, KiwiRail replied:

   In response to recommendation to 011/18 KiwiRail confirms that it has reviewed the system for training and ongoing performance monitoring for the signal box operators and has a programme of changes underway which are scheduled to be in place by the end of 2018.

1. When changes are made to safety-critical systems, the new systems should be fully tested for correct functionality, and the users of the systems should be trained in and familiar with them.
2. Personnel controlling the movement of rail traffic should follow procedures rather than make assumptions regarding the status of signalling equipment.