Report: Richmond water plant didn’t have copy of emergency manual during Jan. 6 failure

Richmond’s water treatment plant didn’t have a physical copy of the document meant to help staff respond to emergencies on the day the facility failed last month, according to a preliminary report issued Thursday by the engineering firm the city hired to review the water crisis.

Employees who work at the water treatment plant also told investigators the city lacks a variety of written materials dealing with training and operating procedures.

“Based on staff interviews, there are no established training procedures or written training manuals at WTP,” the 15-page report from the HNTB firm said. “While on-the-job training is irreplaceable, written training manuals and job descriptions that are updated regularly are critical to ensure every staff member has access to the same information and critical plant knowledge is not lost as staff retire and move on.”

City employees also told the firm that documents outlining plant operations are often “over a decade old” and stored in a way that plant operators wouldn’t know where to find them.

The report says the city’s Department of Public Utilities has an “Emergency Operations Manual” for the water plant that was last updated in 2021. But there was no copy of the manual at the plant “prior to or during” the Jan. 6 failure.

“This plan also lacks facility and process-specific actions that operators of the WTP would need to take in the event of a power outage, which is common of waterworks of DPU’s size,” the report said. “Staff also noted that they had never been involved in any tabletop exercises to simulate a response to emergency situations.”

The report notes that Virginia regulations require waterworks operators to maintain emergency plans for power outages and that those plans be “kept current and readily accessible in the event of a power outage.”

Timeline of events

Though the preliminary report doesn’t offer a full account of what happened on Jan. 6, it provides the most detailed timeline yet of what occurred at the Douglasdale Road water facility when things started to go wrong that morning. The plant’s failure ultimately led to the city being without usable water for nearly a week and also disrupted water service in some nearby counties.

HNTB is expected to provide the city with a final report in a few weeks. The preliminary report, the company said, was based on a site visit to the water plant and interviews with more than a dozen DPU employees. The firm said it has also received documents from DPU, but is waiting to receive additional documents that have been requested but not yet turned over.

The problems at the plant began as early as 4:25 a.m., when a brief power outage lasting a few seconds disrupted the operation of a pump. Then, at 5:45 a.m., the plant completely lost power from its main Dominion feed — “likely due to icing,” HNTB found — taking down the critical information technology network known as the SCADA system that controls all of the facility’s operations. 

Within 10 minutes of the outage, major flooding had occurred, with operators observing at least 6 feet of water in the basements. HNTB writes that without power, plant operators were “not able” to close the valves or pumps controlling the flow of water in the facility. Diesel pumps brought in to help stem the waters also weren’t able to keep up with the incoming flow.

Dominion was not notified of the power loss until 6:45 a.m., a full hour after the outage. 

Several power backups existed at the plant, but none of them immediately resolved the situation. 

Generators were present but had to be manually operated; the report notes that a capital project intended to have them kick in automatically is underway. HNTB didn’t elaborate on the question of why the generators weren’t used on Jan. 6. 

A switch that was supposed to automatically transfer the plant to a second Dominion power source when the first one went out also failed to work. Between 6:40 a.m. and 7:05 a.m., an electrical supervisor struggled to do the switchover manually but encountered a breaker that would not close. 

Although power was restored at 7:05 a.m., the SCADA system failed to reconnect, and workers did not begin closing the valves controlling the movement of water until 7:40. By that time, “equipment was damaged,” HNTB found. 

Summer vs. winter mode

The report also recommended that the water plant be kept in a more durable “summer mode” when winter storms are approaching. In that status, the report says, the plant can draw power from both Dominion feeds instead of just one. The “winter mode” was a cost-saving measure DPU implemented decades ago “during times of the year that power outages from thunderstorms are significantly less frequent.”

The plant should be run on “summer mode” all the time, the report recommended, or “at least during storm events that have risks of power outages.”

In a statement issued along with the report’s release, Mayor Danny Avula said the city is already working to implement many of the changes recommended by HNTB.

“Since water service was restored, the city has made a series of changes to plant operations and equipment,” Avula said. “That includes millions of dollars in investments towards the recovery work, much of which has been spent on equipment upgrades and repairs — including new backup power systems. We have also made meaningful changes to plant procedures, consistent with several HNTB recommendations, as part of continuing efforts to make water treatment plant operations more resilient.” 

In the city’s news release, Interim DPU Director Scott Morris said steps have been taken to better train and prepare plant employees.

“That includes readiness preparation for inclement weather events, as well as regular checks of facility equipment, fuel, chemicals and systems to avert the potential for future impact on operational continuity and service delivery,” Morris said.

(Dominion Energy is a sponsor of The Richmonder, but did not influence or review this story.)