The best stress test

Every two years, Minnkota Power Cooperative invites physical and digital mayhem into its offices. Cellphones are buzzing, emails are piling up and emergency response plans are put to the ultimate stress test during GridEx – North America’s largest grid security exercise.

On Nov. 14-15, about 90 employees from Minnkota and its member cooperatives were scrambling to respond to a barrage of simulated attacks on the electric grid and the personnel responsible for keeping it safe.

“It’s going to feel like the world is falling apart,” warned Dan Inman, Minnkota Vice President and Chief Information Security Officer, at the beginning of the exercise. “That’s by design. We want to test our processes, plans and procedures in a worst-case scenario to find out what we’re doing well and where there are areas for us to improve.”

It didn’t take long for the group to face adversity. Soon, a letter arrived with a suspicious substance, ransomware attacks disabled IT networks and gunfire began targeting field employees and critical energy infrastructure. Power outages spread almost as quickly as social media threats and misinformation.

With so much happening all at once across the entire organization, the only way forward is by working as team.

“In GridEx, it’s neat to see how all of our different departments and business units communicate, especially in a crisis situation,” said Theresa Allard, Minnkota’s compliance manager and a member of the GridEx planning team. “People become accustomed to their own world and what they’re good at, but in reality, it’s all of these moving parts that have to work together to get things accomplished.”

While successes were recognized, identifying shortcomings and process failures in a judgment-free environment is a crucial aspect of going through GridEx. As the scenarios continue to build with new information, it’s difficult for even the most prepared entity to navigate the situation perfectly.

“Of course, when you start throwing everything at them at once, people tend to get a little frazzled, but that’s part of the game as well,” Allard said. “We’re trying to get people to experience that feeling of being overwhelmed and slightly panicked.”

Co-op camaraderie

Minnkota’s plans for GridEx started in January and were built throughout the year. The exercise is designed at a high-level by the North American Electric Reliability Corporation (NERC) through its Electricity Information Sharing and Analysis Center (E-ISAC) for the 250 participating organizations. Minnkota personnel customize information to help test specific plans, processes and procedures.

It was the third GridEx for Minnkota employees as full participants in the exercise, but the first time with the inclusion of representatives from the Minnkota member cooperatives. About 30 member cooperative employees made their GridEx debut this year.

“We really didn’t know what to expect at the beginning,” said Rich Whitcomb, CEO of Red River Valley Co-op Power. “What was interesting about the whole exercise was that each individual injection has actually happened somewhere before in real life. Then, as the events compounded, it really allowed us to analyze our current preparedness plans and find out where we had good sections to work off of and where we were coming up short.”

Fellow first-time player Jeremy Seibel, Beltrami Electric Cooperative’s IT manager, felt the stress of the scenario as his digital infrastructure was bombarded with different attacks and threats. Collaboration with his co-op colleagues helped provide real-life context to improve his existing plans.

“It’s always great to hear another person’s point of view or thought process as well as ideas you might not have thought of on your own,” Seibel said. “Efforts that are important to us might not be as important to a different department or vice versa, so being able to blend those ideas together makes our cooperative efforts and response stronger overall.”

In addition to the member cooperatives, Minnkota personnel also coordinated with the Midcontinent Independent System Operator (MISO) during the exercise and had an observer onsite from the North Dakota State and Local Intelligence Center.

“Every year that we participate, we’re taking that next step to try and make it more realistic,” said Brandon Trontvet, Minnkota System Operations and Energy Management System Manager and GridEx planner. “One major thing this year was bringing in the member cooperatives. It was wonderful to have them here and really strengthen those relationships. It adds value when we’re able to learn from each other.”

Continuous improvement

Throughout the exercise, players documented their actions, observed strengths and concerns. The exercise planners maintained a 26-foot-long timeline with brightly colored sticky notes, each of which captured challenges and responses from the players. The timeline was reviewed during hot washes, a period of time when all players gather to discuss the current state of the game.

While most of Minnkota’s version of GridEx occurred in near-real time, the final section of the exercise pushed the timeline ahead a week and required players to access the damage and long-term restoration schedules. Being the scenario impacts the entire North American grid, existing supply chain challenges and material shortages force players to think creatively about alternatives for the next several months – if not years.

Early in 2024, an after action report will be completed documenting how Minnkota and its members take the lessons learned from this iteration of GridEx and apply them to real life.

“We’re still digesting the takeaways,” Allard said. “Our initial reaction is that we’re pretty prepared for a wide variety of scenarios. It’s really impressive to see how everybody was able to react and generally do the right thing.”

MAIN IMAGE: Dan Inman, Minnkota Vice President and Chief Information Security Officer, warned the 90 GridEx players at Minnkota headquarters that they should be prepared to feel some stress during the exercise.