Case Study – Closed Bus – Knowing the Risks

1. Overview of event

A blackout and subsequent failure of the automatic blackout recovery systems occurred on an offshore construction vessel.  The DP system was designed around a three-way split in the redundancy concept which had been certified and tested for closed bus operation by the responsible classification society.  At the time of the incident, the vessel was operating in a closed ring configuration with one generator connected to each bus.

The vessel’s blackout recovery system was tested on a regular basis.  During these tests, it was common to have all six thrusters back into DP Control within 25-30 seconds of a full blackout.  Individual bus sections were also tested on a regular basis in order to confirm the recovery of a single blacked out bus (redundancy group).

Prior to the incident, the vessel had been experiencing intermittent neutral to ground overvoltage alarms on one of the offline generators which occurred immediately after the generator was disconnected from the bus and running offline.  The event investigation was able to establish a link between the alarms and the effects of one pole of a generator vacuum circuit breaker pole conducting voltage onto the bus (breaking down) while open.  The protection system detected the neutral over voltage excursion caused by the faulty circuit breaker and the generator breaker was locked out from reconnection by the protection system, but this action alone was not sufficient to prevent the fault escalating.  The situation escalated to a blackout and subsequent failed automatic blackout recovery as follows:

• The faulty generator was disconnected and left running offline.  The faulty generator was kept running at rated speed and voltage in an attempt to reconnect to the bus.  The control system deemed this generator as healthy even though its protection relay tripped and latched on the neutral overvoltage alarm.
• During this time, the breakdown across the faulty generator circuit breaker single pole caused a steady-state overvoltage condition on the other two healthy phases, saturating the ferro-magnetic core of the two generator Voltage Transformers (VTs) connected to these phases.  The sustained operation in the saturation region caused the sequential failures of the two VTs.
• The second VT failure, which occurred while the two phases were in a steady-state overvoltage condition, caused a series of voltage surges that exposed the entire 11kV bus through the conducting faulty circuit breaker pole.
• These voltage surges travelled through all connected generators’ VTs as well as to all bus VTs to all voltage sensing circuits within all switchboards in the ring bus.  Elements of these control circuits failed, causing an over-current condition which tripped the over-current protection on all the switchboards VTs.  This caused all generators and thrusters to trip, disabling automatic blackout recovery and manual control over the switchboards.
• The vessel drifted off station.  Manual control over the switchboards was restored by resetting the VT circuit breakers.

2. What can be concluded?

There were unidentified failure modes in this vessel design that could cause a blackout and disable both automatic recovery and manual control.

• The generator protection relays identified the condition caused by the single pole break down failure in the faulty generator, but the action associated with this condition was not effective in isolating the failure and prevent its propagation.
• There were unidentified failure modes in this vessel design that could cause a blackout and disable both automatic recovery and manual control.
• The damaged controllers for the switchboards could not withstand the over voltages created by this type of failure.

3. How can lessons learned from this event be applied to prevent future DP incidents?

• It cannot be assumed that faults detected by the protection system on a generator running offline will be confined to that generator even though its circuit breaker is open.  Shutting down a faulty generator will eliminate the risk of fault transfer associated with single pole breakdown in a vacuum circuit breaker, but other mitigating measures may be required to limit fault propagation from all possible causes (example automatically opening the bus ties).
• The main bus ties should be opened and remain open until the true nature of the fault is understood and rectified (if in doubt open the bus ties).
• All parts of a DP system power plant should be rated to withstand the maximum failure effects and conditions that can be experienced during a fault.  The maximum conditions should be calculated as part of the design process and the equipment rating selected accordingly or measures to prevent overvoltage reaching damaging levels included in the design, which should be proved through testing.
• Grounding system must be designed to limit steady-state voltage escalation and to effectively dampen voltage oscillations caused by intermittent faults.
• Consideration to install high speed data logger on 11kV power plant protection and control system.

4. Relevant guidance

The following IMCA Guidance would be relevant to this case study:

• IMCA M 220 – Guidance on Operational Planning
• IMCA M 103 – Guidance for the Design and Operation of Dynamically Positioned Vessels
• IMCA M 166 – Guidance on Failure Modes and Effects Analysis (FMEA)