Power and communications: two infrastructures intertwined
Designing utility communication networks for increased energy resiliency
For most of us, when the electricity stops, our lives grind to a halt. At home, it means no lights, no appliances, no air-conditioning or heat, and — the amenity likely to get your attention if the others do not — no Wi-Fi. As we transition to a low-carbon future, most Americans may be unable to drive or cook if the electricity does not flow. Outside our homes, the effect of a black-out is just as impactful: traffic lights go dark, supermarket freezers fail, and industrial activities are paused while all of our critical services fall back on generators. The resilience of the electricity sector is fundamental to the wider resilience of our communities and cities.
Most people will agree that their phone and its connection to the internet are used in almost every daily activity and that our communications infrastructure is equally essential to our nation’s resilience. In fact, with neither electricity nor internet, it is hard to think of any functions that would continue unaffected in their absence; particularly in the world towards which we are moving in the coming decades.
As we look towards the future evolution of these two essential infrastructures, we must first look back on how they have evolved over the decades. It may not surprise you that rapid progress in information and communications technologies (ICT) have been adopted and incorporated into the infrastructure of the electric grid; the result being that the electric and communication sectors have become co-dependent, as illustrated in Figure 1.
Terminology toolbox:
TDM: Time-Division Multiplexing is a method for transmitting and receiving many data streams over a common signal path in a rigid, deterministic manner.
SONET: Synchronous Optical NETwork is a protocol that enabled easy multiplexing and demultiplexing of a high number of multiple data streams over long distances through fiber optic cables.
TCP/IP: Transmission Control Protocol / Internet Protocol are the foundational technology that enabled scalable interoperability and upon which the internet is based.
MPLS: Multi-Protocol Label Switching is a flexible way to build reliable connections for real-time communications and accommodates a wide variety of traffic types (voice, video, and data).
LTE: Long-Term Evolution is a broadband wireless communications standard for mobile devices and data terminals that has achieved worldwide adoption. LTE is a fourth generation (4G) cellular telephone standard.
Utilities are moving towards third party communication networks as grid modernization is underway
Communication Service Providers (CSPs) like AT&T, CenturyLink and Verizon provide internet and telephone services to millions of residential customers, but also support businesses, including electric utilities. While many utilities rely to a great extent on their own private networks, there has been an increasing trend toward using CSPs as part of utility operational networks. This is primarily motivated by the ready availability of CSP networks and services, and the urgent need to set up communications for expanding grid operations.
This urgency is somewhat driven by the massive increase in IoT devices coming online in recent years, especially in the distribution domain. CSPs may provide adequate security and reliability for most utility applications, and are typically quicker and easier to buy than to build using the utility’s private infrastructure.
Considering the increasing reliance of electric utilities on CSPs, and society’s increasing reliance on electricity, resilient communications for the supply of energy has become a focal point of concern. That concern encompasses fortifying against potentially catastrophic events.
Nashville bombing on Christmas Day 2020
Readers who are based in Nashville, Tennessee, will certainly recall the bombing that took place on Christmas Day 2020. The explosion damaged an AT&T communications facility that not only provided local telephone, internet and wireless services, but also served as a critical node for regional connectivity. The damage to the facility wiped out its commercial power supply, but the facility ran on backup batteries for hours after the blast. These backup batteries and generators were soon damaged by fire and/or water which resulted in communications outages across Tennessee and neighboring states Kentucky, and Alabama. Restoration efforts focused on connecting external backup generators to power critical equipment. Meanwhile, teams from AT&T worked to deploy portable cell sites in the area and reroute regional network traffic while the local utility focused on restoring permanent power to the critical communications building. By that Sunday, the 27th, AT&T reported that 96% of wireless network, 60% of business services and 86% of consumer broadband and entertainment had been restored .
Co-dependent critical services
The events in Nashville highlight how reliant communication services are on power, and how back-up generators can sometimes fail. Knowing that many operations in the power grid are increasingly relying on communication networks reveals the potential for closed loop dependencies between power and communications infrastructure. For instance, the impact of an event that causes power blackouts to comms facilities may be compounded where that utility relies on that CSP for its normal operations.
Of course, utilities prepare for any so-called “Black Sky” event where they must restore power to the grid without their normal communications in operation. The industry defines “Black Sky” hazards as catastrophic events that result in multiple system failures, potentially across large regions for extended periods of time. Such events include terrorist attacks and geomagnetic disturbances caused by solar storms, among many other examples. Characteristic of “Black Sky” hazards is that they typically impact both electric and communication sectors, so the co-dependency of these infrastructure systems plays a role in utilities’ preparations. Ongoing research efforts focus on fortifying infrastructure so that it can provide service in challenging conditions; work that goes beyond optimizing the grid for business-as-usual operations.
Utility efforts for resilient communications
Let us take a look at some of the measures taken by utilities aimed at maintaining resilient communications.
- The majority of electric sector Protection & Control (P&C) systems remain largely independent of CSP networks. Fundamentally, P&C systems allow the greater part of the grid to remain in operation when a fault occurs by isolating that branch of the grid during restoration. For example, one category of P&C systems, teleprotection, is responsible for fault detection which allows coordinated tripping to quickly isolate faults.
- SCADA (Supervisory Control and Data Acquisition) provides real-time monitoring of the grid and is used in both transmission and distribution. As utilities consider both teleprotection and SCADA as critical to operating the grid, and recovering it during a fault, utilities often prefer to host them on private network communications infrastructure. With copper retirement over the last decade (see Figure 1), many utilities that used CSP “private line services” for SCADA are being forced to replace CSP services with alternative solutions.
- Private utility communications networks use technologies such as dedicated fiber optic routes attached to overhead power line structures or placed in dedicated underground conduits, which are aimed at enhancing reliability.
- Also employed in large numbers are point to point microwave radio systems and point to multi-point private radio networks.
- LTE wireless networks are another considered option to increase the resilience of power sector communications during mutual aid events. A growing group of utilities is investing in private LTE networks that operate in an exclusively licensed spectrum: because spectrum is a limited resource not unlike real estate, however, it is costly. Industry efforts to prepare and mitigate the effects of “Black Sky” hazards are ongoing and private fiber networks remain an attractive option in their potential to effectively diagnose and recover the grid in the aftermath of a “Black Sky” event.
Creating a wide-scale utility fiber network
For resiliency in the face of disastrous events, widescale interconnection of private fiber networks is an attractive solution for utilities. To create such a network, however, requires the allocation of resources to plan and protect against events that may never have happened before or have occurred in such low frequency that it is difficult to quantify their risk or re-occurrence; events known as ‘high-impact low-frequency’ (HILF). As identified in the NAERM project, understanding how the systems work and where the interdependencies lie is a necessary first step before effective decision making can be done to pinpoint where investments should be made to improve energy resilience.
Creation of a wide-scale utility fiber network that interconnects and fills gaps in existing utility fiber routes may overcome issues of interdependency. The first step in this direction is the assessment of the investment required. Practically, this means sharing of information about the locations and characteristics of existing utility fiber optic network infrastructure, though lack of clarity on the application of NERC CIP regulations to this type of information has complicated collection efforts to some degree.
Assuming adequate collection of this information, the step by step process to complete the feasibility assessment includes;
Step 1: Determining the extent of existing infrastructure that can be leveraged by interconnecting fiber at existing points of possible interconnection (e.g. transmission substations).
Step 2: Estimating the number of current gaps (and their average length) that would need to be filled to provide a useful national communications network for the electric sector.
Step 3: Creating a high-level cost estimate for the construction of fiber links to eliminate the identified gaps. Stakeholders may utilize these cost estimates to inform their decision making in furtherance of their efforts to enhance system resilience.
The path forward
The power system has always been designed with redundancy in mind. In today’s world, the same design principles are being focused on fortifying the communications networks on which the grid relies, and on which its reliance will likely continue to increase as the smart grid of the future develops. Still, while resilience is perceived as essential, it is by no means straightforward. Utility efforts to build mission-critical communications infrastructure is complemented by collaborative efforts between national labs, regulators and industry members. Obtaining a better understanding of infrastructure co-dependencies is a first step towards building resilient utility communications and is foundational to supporting broad grid modernization efforts.
EPRI’s Program 161 Information and Communication Technology portfolio page highlights some research activities including;
- Program research project 161G: Telecommunications
- Public report: Resilient Communication Demonstration Project
- Supplemental project: Long-Term Evolution (LTE) Security
- Supplemental project: High-Altitude Electromagnetic Pulse (HEMP) E1 Hardening of Bulk Electric Systems (BES) Communication Systems
Join the discussion by posting in the comments section and reach out if you want to hear more about EPRI’s research in these areas!
