An ‘incomprehensible failure’

“Before implementing Phoenix, Phoenix executives did not ensure that it could properly process pay. When the system was put in place, it could not perform some critical pay functions, such as processing requests for retroactive pay.”

“The department knew about many of these critical weaknesses before implementing the Phoenix system. In our opinion, these weaknesses were serious enough that the system should not have been implemented.”

These comments come from a recently released report by Canada’s auditor general Michael Ferguson, describing how the federal government failed to properly manage its plan to implement a new pay system for its 290,000 employees.

It is his second report on the troubled Phoenix project, which the government began in 2009 and implemented in two waves in 2016.

The report is critical of decisions taken by the project’s executives, saying: “The Phoenix pay system is less efficient and less cost-effective than the old system, and thousands of employees have not been accurately paid or paid on time.”

It added that project mismanagement has resulted in the government having to spend a “significant amount of time and money” trying to fix problems.

While the report is about failures within Public Services and Procurement Canada (PSPC) — the department responsible for Phoenix — it can provide lessons for any employer planning to redesign its payroll system.

One of the areas of concern raised in the report was that Phoenix executives did not fully consult other government departments and agencies to find out what they needed from a new pay system.

“(PSPC) did not share a complete list of functions with departments and agencies and did not give them a chance to review or approve the functions to confirm that the system met their needs,” said the report.

When Phoenix executives were faced with higher than expected costs for developing and implementing the pay system, they chose to remove or delay implementing some pay processing functions rather than asking the Treasury Board for more money.

“Phoenix executives decided to defer or remove more than 100 important pay processing functions, including the ability to process requests for retroactive pay; notify employees by email of actions required on their part to process pay requests; and automatically calculate certain types of pay, such as increases in pay for acting appointments,” the report said.

To meet timelines, the report said project leaders opted not to fully test or pilot Phoenix before implementing it, limiting their ability to know whether it would work as planned.

“This pilot was the department’s chance to test a final, live version of Phoenix before implementation. The pilot could have allowed the department to detect problems that would have shown that the system was not ready,” said the report.

The report also criticized PSPC for failing to plan for future upgrades to the software application on which Phoenix was built and for not having a thorough contingency plan to deal with possible problems.

“The plan mostly outlined what to do if Phoenix failed to operate; the plan did not anticipate scenarios with system or process problems, such as the ones that occurred after implementation,” the report said.

It also criticized PSPC for not complying with government policies for security, privacy protection, and accessibility.

“The department implemented Phoenix despite knowing about high security risks, which it did not expect to address until December 2016 — months after the planned implementation. These risks included the risk of someone gaining unauthorized access to information.”

The report said PSPC failed to complete a final privacy impact assessment before launching Phoenix.

“The privacy commissioner of Canada has reported numerous privacy breaches of federal employees’ information in Phoenix after it was put in place,” it said.

The report also noted that in 2013, Phoenix executives eliminated any obligation for IBM — which was responsible for installing the Phoenix software — to customize it to make it more accessible for people with disabilities.

“After Phoenix was put in place, federal employees with disabilities or other impediments reported difficulties reading Phoenix information.”

It was also critical of executives for not listening to warnings that the government’s pay staff was not ready for the new pay system to be implemented.

Besides replacing the government’s 40-year-old pay system, the Phoenix project involved centralizing payroll for 46 of the government’s 101 departments and agencies at one location in Miramichi, N.B., and reducing the number of staff. Previously, each department and agency had its own payroll professionals.

Although PSPC believed the changes would make pay processes more efficient, the report said they had the opposite effect.

Previously, each pay adviser in the departments and agencies handled about 184 employee pay files. With centralization, PSPC expected the number to rise to 200 and then to at least 400 after launching Phoenix. This would allow the pay centre to handle an initial 92,000 employee pay files from the 46 departments and agencies whose pay operations the government had centralized.

However, in July 2015 — seven months before the rollout would begin — the report said PSPC knew Miramichi employees could only handle about 150 files each, for a total of 69,000 files.

The report attributed the lower amount to inexperience and lack of training, given that many pay advisors who had worked in the departments and agencies did not move to Miramichi and the government had to hire new staff.

“Despite knowing that the Miramichi pay centre could handle 23,000 fewer files than expected before the implementation of Phoenix, Phoenix executives did not re-examine expected benefits,” said the report.

“Phoenix executives also did not adjust the number of employee pay files that pay advisors were expected to handle or did not hire more pay advisors.”

They also didn’t do enough to help departments and agencies adjust. While PSPC prepared a training plan, curriculum and procedures manuals, the report said officials from the departments and agencies that took part in the audit said their employees did not get adequate training.

“The content and procedures were generic, incomplete, and used classroom or web-based presentations instead of a demonstration version of the Phoenix software,” the report said.

And there were other red flags that Phoenix was not ready.

In December 2015, after concerns from officials in departments and agencies moving to Phoenix, the Treasury Board hired IT consulting company Gartner to assess how ready they were for the change.

Gartner’s report, presented two weeks before the first Phoenix rollout in February 2016, stated that the new pay system might not be able to pay employees accurately or on time because it had not been fully tested and there may not be enough time to fix problems before launching.

Gartner recommended that PSPC implement Phoenix gradually, beginning with departments and agencies with less complex payrolls. It also advised that Phoenix and the old system run in parallel in case of issues. The audit report said PSPC “did not consider the report’s findings and recommendations before Phoenix was implemented.”

The audit also criticized PSPC for not having independent oversight on the launch decision. The only information that the PSPC deputy minister received about Phoenix was from Phoenix executives.

“When Phoenix executives briefed the deputy minister just before Phoenix was implemented, they did not provide important information about problems with the system,” the report said. “The lack of oversight allowed Phoenix executives to implement the system despite clear warnings that the Miramichi pay centre and departments and agencies were not ready and that the pay system had significant problems.”

“If there had been effective oversight, the deputy minister (of PSPC) would have received complete and accurate information on Phoenix readiness. This could have resulted in a different decision to implement the system,” it said.

In response to the report, PSPC said it was making improvements to the way it manages projects and strengthening its policies and governance structures to ensure the Phoenix mistakes are not repeated.