As water plant crisis began, Richmond officials scrambled to fix issue at DPU call center

As water plant crisis began, Richmond officials scrambled to fix issue at DPU call center

On the same day it had to announce Richmond’s tap water was unsafe to use without boiling it, the city’s Department of Public Utilities had a different communications problem.

On the morning of Jan. 6 — after a snowstorm had caused Mayor Danny Avula to declare a state of emergency the day before and close city government offices — DPU officials discovered the department didn’t have a phone message in place for its call center telling customers the office was closed for the day.

“Good morning. The closed message is not on the for the non-emergency call center,” DPU Deputy Director Lynette Lemon said in a text message to former DPU director April Bingham a little after 8 a.m.

Lemon said DPU was trying to reach the city’s Department of Information Technology for assistance.

“We are trying to reach DIT?” Bingham texted back. “Who in DIT?”

The back-and-forth over the phone message was revealed in communications records The Richmonder obtained through a Freedom of Information Act request seeking emails and text messages on the day the water crisis began.

It doesn’t appear the phone message snafu — which was happening as officials were trying to assess the scale of the trouble developing at the water treatment plant —  significantly impacted city operations that day. It was resolved that morning, long before residents would have noticed water pressure issues and called DPU to ask about it.

But the phone problem shows that — despite the emergency declaration meant to bring a heightened state of alertness and preparation — even a basic task involving communication with city residents briefly slipped through the cracks. Bingham had to divert at least some of her attention, the records show, to help troubleshoot the missing phone message as the water crisis was beginning.

The utilities department has a single phone number customers can use for both emergency and routine calls. When city offices are closed, according to DPU, callers to that line typically hear a closed message but can still choose an emergency option to get immediate assistance.

A little before 10 p.m. on the Sunday night prior to the water plant failure, a DPU staffer sent an email to a total of eight city employees across DPU and IT asking for confirmation the closed message would play the next day. That email was marked “high” importance, but there was no reply that night.

The next email in the chain came at 7:12 a.m., when another DPU official again asked the IT department for a status update.

About 45 minutes later, an IT staffer said he had “opened a ticket” to get the problem resolved.

Another half-hour passed. At 8:32 a.m. Bingham herself emailed the IT department asking for assistance. At virtually the same time, she was exchanging text messages within her own agency about what had gone wrong at the water plant.

At 8:40 a.m., an IT staffer said he had activated the closed message and called the number to confirm it was on.

Other communication issues

By that afternoon, DPU was starting to hear from customers having water problems.

A Virginia Commonwealth University official inquired about water before the city had released anything publicly about problems at the plant.

“Has the city lost water?” VCU Associate Vice President for Facilities Management Rich Sliwoski asked in an email to Bingham at 2:53 p.m.

Around 3:30 p.m., one resident emailed Bingham and DPU customer service saying they and their neighbors in the Fan District were “experiencing lower water pressure and would appreciate your help.”

Another resident put it more succinctly, emailing to say: “We have no water.”

It’s unclear exactly when DPU’s emergency call center agents — who would presumably have been fielding calls from residents noticing their water disappearing — were told there was a broader problem with water service. One email suggests it happened mid-afternoon.

At 3:20 p.m., DPU Senior Program and Operations Supervisor Lisa Smith sent an email saying call center agents should stop scheduling service orders because the “pumping station is currently down which affects our water supply.” That was followed by a second email saying at least one water zone was “totally down” and others were losing pressure.

“Do we know if communications will send a notice to customers?” a DPU staffer replied at 3:24 p.m, saying the customer service team was already “seeing submissions” about low water pressure. “We want to be consistent in our messaging/response.”

The city put out a news release on the water problem a little before 4:30 p.m., urging residents to conserve water and boil it before use.

When asked to explain the delay in communicating with the public, officials have said they felt there was a chance the problems at the plant could be fixed quickly enough there would be no disruptions to water service and no need for a boil water advisory. It’s not clear why the city’s initial read on the situation was overly optimistic, but that question will likely be explored further in a pending third-party review of the incident.

Though city officials appeared slow to grasp and broadly communicate the magnitude of the problem, Bingham and water plant Superintendent Doug Towne were both aware of power issues at the plant prior to 7 a.m., according to their text message history.

Bingham was up particularly early as officials assessed the snow impacts, participating in a 4 a.m. update on how the city was faring.

DPU tracks how its call center is performing, and one of those updates came in on the morning of Jan. 6. The center received 1,940 calls in the first few days of the month, with 1,421 going to the non-emergency line.

Officials were enthusiastic about stats showing 98% of calls had been answered in January with an average speed of answer at 44 seconds.

That was an improvement from December, when 92% of calls were answered at an average answer speed of a little more than two and a half minutes.

City offices were closed again the day after the water plant failure. On the night of Jan. 6, DPU and DIT officials confirmed the call center’s closed message would play correctly on Jan. 7.