- Enter the IMCA Awards 2024: We\'re looking for outstanding Safety, Sustainability, and GHG projects.
- IP Code: Mandatory Code enters into force on 1 July. Make sure your company is up to speed with the changes.
- Early Bird tickets available for the IMCA Global Summit 2024. 3-4 December, Utrecht, Netherlands. Members can save £400.
- Discover how we are helping Members reduce emissions: Explore the ProjectGHG hub now.
- Now booking. Join the Lifting and Rigging community at our Technical Seminar in Amsterdam. Thursday 12 September 2024
- Now booking. Underwater Ship Husbandry Seminar. 10 October 2024, Singapore.
Software update caused loss of generators in separate redundant groups
Case narrative
A DP2 platform supply vessel (PSV) was set up on auto DP whilst conducting trials in open water after extensive upgrades. The redundancy concept was based on two redundant groups with power generation, thruster supplies, and all DP consumers equally split between them. Bow thruster 1 was supplied from a dual drive which can be supplied from either redundant group. Each redundant group has dissimilar sized generators as shown in the graphic below.
The vessel is designed to operate with open bus ties at all power levels. At the time of the incident, the main bus tie was open, separating the two redundant groups. All generators were online. The weather was fair and sea conditions were light with wind speed of 1 knot, current 2 knots and wave height of 1m.
The vessel was performing a turn when the DG2 breaker unexpectedly tripped, followed by DG4 breaker tripping a few seconds later. The DP operator received alarms indicating loss of the generators and engineers observed alarms for loss of generator protections in the engine control room. The vessel was able to maintain position due to the light sea conditions, but power limitations became active for thrusters in the port redundant group.
Subsequent investigation identified that DG2 breaker had tripped due to a defective Paralleling and Protection Unit (PPU). The PPU is a microprocessor-based protection unit containing all of the functions for protecting the generator. The PPU is set up to trip the generator in the case of over-current, under-voltage, short circuit, reverse power and excitation problems. It is important to note that a generator breaker will also trip if it’s PPU loses power or reboots.
As well as generator protection, the PPUs were configured for load sharing, where the PPU will maintain the rated speed of the engine following an increase or decrease in generator load. The interface between the PPUs for active load sharing was via a ‘daisy chained’ CANBUS network as shown in the graphic below.
Further tests demonstrated that switching the power supply of DG2 PPU on and off would cause the screens of other PPUs to flash and reboot. This was observed to occur approximately fifty percent of the time. The flashing and reboot would cause the respective generator breaker to trip. It was noted that cycling the power of DG2 PPU could cause any other generator to trip, regardless of redundant group.
Analysis of the faulty DG2 PPU found that it had a defective power supply and a software error. An intermittent power supply fault had caused DG2 PPU to reboot during operation. A ‘bug’ in the software caused the PPU to boot up with the wrong communication settings and data was sent at the wrong speed to other PPUs over the CANBUS. Receiving data at the wrong speed caused errors in the other units on the CANBUS, causing breaker tripping. The root cause was a coding error in the boot loader software for the PPUs.
The defective software version had been installed by switchboard vendors during recent vessel upgrade work. The software updates were produced by the PPU manufacturer and downloaded from the internet. The switchboard vendors assumed that using the latest software version was safe and it was their company policy to update the software anytime work was done on the units.
The lessons
- Initial fault finding was hampered because the vendors had left no record of installing different software versions. A fleetwide review showed that the faulty software had been installed on other vessels. After the incident, software versions were recorded and documented in the FMEA.
- The control of software and those vendors involved with software needs to be managed as part of a formal management of change (MoC) process and the need/benefit of the version fully understood prior to its acceptance.
- Although testing of the PPU had been undertaken during FMEA proving trials, the hidden failure was only introduced during the software upgrade. The design of the CANBUS is being further analysed to determine the need for all PPUs to be connected together, rather than just those within each redundant group. The results of this analysis may allow the splitting of the CANBUS and the removal of any potential hidden failure in this system in the future.
This case study demonstrates the importance of software MoC on vessels. Although it is accepted that the best of MoC processes may not capture issues such as a bug left in the software by the producer of the software. In this case having such a process would have significantly aided the identification of the issue.
The best form of prevention of such a fault is not having unnecessary network links across redundant groups in the first place. Such that any maloperation of software cannot propagate between one redundant group to another.
DP Event
Published: 13 June 2019
Download: IMCA DPE 02/19
Submit a Report
The following case studies and observations have been compiled from information received by IMCA. All vessel, client, and operational data has been removed from the narrative to ensure anonymity.
Case studies are not intended as guidance on the safe conduct of operations, but rather to assist vessel managers, DP operators and DP technical crew in appropriately determining how to safely conduct their own operations. Any queries should be directed to IMCA at [email protected]. Members and non-members alike are welcome to contact IMCA if they have experienced DP events which can be shared anonymously with the DP industry.
IMCA’s store terms and conditions (https://www.imca-int.com/legal-notices/terms/) apply to all downloads from IMCA’s website, including this document.
IMCA makes every effort to ensure the accuracy and reliability of the data contained in the documents it publishes, but IMCA shall not be liable for any guidance and/or recommendation and/or statement herein contained. The information contained in this document does not fulfil or replace any individual’s or Member's legal, regulatory or other duties or obligations in respect of their operations. Individuals and Members remain solely responsible for the safe, lawful and proper conduct of their operations.