Minnkota shows grit and growth during GridEx

Explosions occur at major power facilities. System failures begin to spread. Rolling blackouts are followed by cyberattacks, security breaches and social media upheaval.

If something could go wrong, it almost certainly did during GridEx VI – a nationwide training exercise held Nov. 16-17 to prepare utilities for cyber and physical security threats. About 50 Minnkota employees participated in the event, which tested their ability to work together and respond to a barrage of nightmare-inducing scenarios.

“GridEx is an excellent opportunity to test the resiliency and reliability of our systems in a worst-case scenario without having to disrupt any of our actual systems,” said Justin Haar, Minnkota cybersecurity specialist. “The nature of the exercise is so much bigger and more complex than most organizations would ever test on their own. It forces us to be creative and responsive versus just conducting a basic tabletop exercise.”

GridEx is organized every two years by the North American Electric Reliability Corporation’s (NERC) Electricity Information Sharing and Analysis Center. As the regulatory watchdog over the nation’s grid, NERC developed the framework and overarching narrative for the 700 participating organizations. Minnkota planners, which included a cross-section of employees, customized scenarios specific to the cooperative’s infrastructure and systems.

This is Minnkota’s second time as an active participant in GridEx. In 2019, cooperative employees gathered in one large room to go through each scenario together. The exercise became more sophisticated in 2021 with most participating employees being distributed into their natural work environments. Exercise updates, called injects, were delivered to predetermined individuals, which tested their ability to not only respond to the issue, but to relay information to others in the organization.

“Impressively, our people knew their roles and how to respond; however, there’s always room for improvement,” said Theresa Allard, Minnkota compliance manager. “We really wanted to test communication and the distributed play allowed us to make the scenarios much more realistic.”

As communication filtered through the organization, war rooms were developed where team members could gather and develop solutions to complex technical problems. All steps taken throughout the exercise were documented so that plans can be improved or resources can be found to address potential risks in a real-life situation.

Bringing departments together

The stressful nature of GridEx is meant to overwhelm even the most prepared utilities and uncover any deficiencies in processes or procedures. But the high-pressure situations also provided an opportunity for camaraderie to develop between departments that do not routinely work together.

“You can see the respect grow when people understand how complicated emergencies are and how many different people are involved in resolving issues,” Allard said.

Problems were not easily solved, as GridEx assumes all utilities are in a state of crisis. That means equipment replacements may not be available, law enforcement may not be immediately responsive and there is likely very little help coming from other organizations.

Minnkota’s issues ranged from catastrophic substation and power plant failures to rolling blackouts and ransomware attacks on the cooperative’s IT infrastructure. As the exercise evolved, these challenges built on each other and created extreme difficulties for employees to navigate.

“It forces us to confront significant problems because the problems aren’t only impacting us, but every other utility in the country,” Haar said, adding that in future GridEx events, the cooperative would like to include its members and other emergency entities.

Managing the message

While groups of technical experts mapped out ways to bring power back to communities in need, the Communications staff crafted messages for internal, external and member audiences. GridEx emphasizes social media and provides a simulator, called SimDeck, to practice making and responding to social media and website posts.

Misinformation was spread on the platform during the exercise from malicious actors who were trying to create chaos. Threatening social media posts were also reported back to Minnkota’s safety and security personnel.

“Oftentimes companies aren’t judged on what happens to them in terms of disasters, but they’re judged on public perception, how the media handles it and how people talk about it,” said Jason Uhlir, Minnkota safety manager. “So there is a crisis in front of us and we have a whole second crisis of public perception. SimDeck gives us an opportunity to see those dynamics play out and address them.”

As one of the planners of the exercise, Uhlir said he’s always impressed with the problem-solving ability of staff and the resourcefulness in finding solutions in a way that the planning committee didn’t predict. As Minnkota’s risks continue to evolve, there is no time for complacency in emergency planning.

“Every time we do GridEx, it’s an opportunity for us to grow and to continuously improve so that if we do find ourselves in a crisis situation, our members aren’t going to be the ones that suffer,” Uhlir said.

MAIN IMAGE: (Left to right) GridEx planners Theresa Allard, Jason Uhlir and Katherine Anagnost work to document processes and procedures. (Minnkota/Michael Hoeft)