‘I’ve never seen it not switch over’: Details emerge of the crucial time period that led to water crisis

Richmond Mayor Danny Avula on Thursday gave the most detailed account yet of how a snow-induced power interruption Monday morning at the city’s water plant led to spiraling problems that caused much of the city to lose water this week.

In a 42-minute call with reporters that focused largely on technical questions, Avula explained that the water treatment plant is connected to two different power sources on different circuits of Dominion’s grid. In the event of a power outage that knocks one circuit out, a switch is supposed to automatically transfer the plant to drawing power from the other circuit in order to keep electricity continuously flowing. 

After an outage knocked out one source at 5:50 a.m. Monday morning, that switch failed to flip on its own. An on-call electrician was summoned to the site and manually connected the plant to the other power source by about 7:30 a.m. 

“The operator I talked to said, ‘You know, I've been here 25 years, I've never seen it not switch over.’ We've been through numerous power failures over the years, and the redundant Dominion source has addressed that,” Avula said, adding that the switch failure was on the city’s side and not Dominion’s.

The electrician who reconnected the plant to Dominion’s power source also determined it wasn’t necessary to power up its generators, which must be turned on manually, because power was flowing to the facility from the electrical grid, Avula said. 

But in the meantime, the interruption of power had triggered another catastrophic failure, this one of the IT system that controls all of the plant operations, known as the supervisory control and data acquisition, or SCADA, system. 

While the manual switchover was underway, the plant was running on a form of battery backup known as an uninterruptible power supply. That form of backup, which is designed to be used for short periods of time to bridge power interruptions, only lasted about 45 minutes. 

“The time to switch over to the second Dominion Power system was a little bit longer than that,” said Avula. “And so the battery backup going out is what led to the computer system, the SCADA system, crashing, and then when we rebooted the SCADA system, we weren't able to connect it to servers.”

The failure of the SCADA system then prevented water valves in the plant from opening, leading to massive flooding that rapidly submerged other critical equipment.

Avula said there were “multiple attempts” to pump the water out of the plant, but “the rate that the basement well was filling exceeded the sump pump capacity.”

At a news conference Thursday, Avula initially indicated he didn’t have enough detail to answer questions on how many employees were staffing the plant during the failure and whether they attempted to manually activate generator power. The mayor and his team later regrouped and attempted to provide more detailed responses in the subsequent call with reporters.

City government was closed Monday due to the weather, but officials have indicated they don’t believe short staffing was a factor in the failure because plant employees are presumably considered essential personnel. Avula said a “full staffing component” was at the plant Sunday night and into Monday morning.

Though the power outage reportedly occurred Monday morning, Avula said at a news conference that he “first heard about this” early Monday afternoon. The mayor said he was told earlier in the day that there were some “power issues” at the plant, but didn’t get a full, formal briefing on the situation until around 1 p.m.

Avula has said the city will prepare a full after-action report on what went wrong, and he pledged those details will be made available to the public.

“It will be laid out and that will be a public document,” he said.

Asked when the city last tested all the backup systems to ensure they were functioning properly, Avula said he couldn’t immediately say but intends to include that information in the forthcoming report.

The mayor indicated he has been more focused on getting through the immediate crisis than gathering precise details of what caused the failure.

“Right now, job number one is get the boil water advisory lifted and then prepare for any potential weather event coming,” Avula said, referencing the forecast for more snow this weekend.

Part of those preparations, Avula said, involve “increasing staffing and surveillance roles at the plant.”

Asked if he still has confidence in Department of Public Utilities Director April Bingham, Avula said he does. But he also said he’d be willing to make staff changes or hold employees accountable if new information emerges that points to management failures, because “that’s the job.”

“I've been absolutely impressed with April's attentiveness and response,” the mayor said. “Again, I’m a week into this job just getting to know everybody, know the operation. … Once we do our after-action review and once we understand exactly what happened, that'll bring a lot to light about how much of this was system failure, how much of this were process failures. And again, I am absolutely committed to identifying what we need to do to prevent this from happening again.”

(Dominion Energy is a sponsor of The Richmonder, but did not influence or review this story.)