Teamwork at its best

If you are an IT professional supporting major production environments and applications, you have most likely experienced a significant system outage at some point. We had one of those events thiscanstockphoto16328410 week. As in previous experiences in other organizations, I saw people at their best come together as a team working diligently to restore systems. This team included IT, clinical and operations staff.

I know CIO colleagues who recently managed through a week long outage of their business systems in one case and a multiple day outage of their electronic health record in another. They could probably share similar lessons following those experiences. 

In the spirit of teaching and learning from one another, I offer these key points if you have a significant event:

        Quickly bring in all required parties to diagnose and resolve the problem

        Quickly assess the impact and engage the right operational leaders

        As you understand the scope and extent of the situation, bring in additional expertise

        Determine the remediation steps needed from a systems and operational perspective

        Establish a communication plan with regular updates to the user community; provide very specific and clear information that staff need in order to do their work during the incident

        If the incident lasts longer than expected, ensure that key staff have backups so they are able to get some rest and do formal handoffs for continuity

        Assemble the right people to monitor the situation and jointly make decisions on recovery taking into account all operational impacts

        As the CIO, own it and provide updates to executives with full transparency

        After the incident, do a full Root Cause Analysis (RCA) of what happened and debrief on how the incident was handled; identify action items, assign the person responsible, establish due dates and track progress

When I described to my husband the incident we had this week, he could relate. Before he became a minister he was the data center manager for a national benefits consulting firm. He recalled major outages they experienced. But he was quick to put them in perspective saying they were just moving money around, not running a hospital and taking care of patients.

Health care is different. We are dealing with people’s lives. Patient safety is top priority. For each checkpoint call during the incident we followed the same framework – which systems are available, which systems are unavailable, what patient safety concerns do we need to address, and a roll call of all hospitals for any specific issues.

In the command center we discussed in detail the operational impacts with clinical leaders. Listening to the clinical leaders’ questions and concerns involving basic workflow and patient care reminds us that what we do matters. As IT professionals, we don’t touch the patient. But we are a critical part of the extended patient care team as the systems we provide and support are depended on by the clinicians and caregivers 24/7.

Our debriefing and RCA from this incident has already begun. There will be opportunities to improve. This will include steps to prevent a similar incident in the future and to improve how we manage such incidents to ensure seamless operations and continuity of care.

As IT professionals, we work hard to ensure outages don’t happen.  But if they do, we come together as a team and deal with it. I’m grateful to our very talented and committed team and honored to be working with them.         

6 thoughts on “Teamwork at its best

  1. Karen Hollingsworth on said:

    Sue, wise words about learning from the experience to improve in the future for certain. I had recent experience on the importance of the communication systems – both paging and phones and we are way to improving. But … also too bad to have to experience… Karen

  2. Keith MacDonald on said:

    It doesn’t get more real-life than this, Sue. Thanks for sharing. Hope things have stabilized for you and the team there.

  3. Rodney Nelson - UMHS/MCIT on said:

    Thank you Sue for posting. Often times we are in a major incident situation, it is important for us as leaders to remember these key points, as it is very easy to dive deep into the details and miss the “big picture”.

    Thank you again for posting!

Leave a Reply