Skip to main content
The state of Oregon has spent $46 million on a new phone system for more than 30,000 state employees in 400 offices. It doesn't work.
On March 12, for instance, a state employee tried to return a call to WW about the new system, but he found his desk phone inoperable—again.
"No dial tone," explained the employee, who spoke on background because he was not authorized to talk about the phones. He was forced to return WW's call on his cell.
Non-working phones have been a regular complaint at many state agencies over the past two years, although the trouble-plagued system has until now received little public attention.
But on Jan. 26, the state issued a formal notice of default to IBM, the contractor that built and installed that new phone system, which is called Project MUSIC. (That stands for "mobilizing unified systems and integrated communications.")
"IBM has failed to provide the state with a stable, reliable managed communications solution," wrote Lori Nordlien, a procurement officer at the state Department of Administrative Services wrote in a letter to the company. "IBM has materially failed to perform."
The default notice, which WW obtained via a public records request, could mark the beginning of legal action and is a sign of how unhappy the state is three years after signing a contract with IBM.
The failure of something as seemingly routine as a phone system harks back to other high-profile state government technology debacles: the scandal-plagued state emergency radio system, known as the Oregon Wireless Interoperability Network, and of course Cover Oregon, the $300 million online health care exchange that failed in 2014, sparking more than two years of bitter legal wrangling with the contractor, software giant Oracle.
Liz Craig, a spokeswoman for the Department of Administrative Services, says Project MUSIC is different from OWIN and Cover Oregon. "First, Project MUSIC is not building anything new. Rather, it is replacing existing hardware and connecting it to existing infrastructure," Craig says. Second, she adds, DAS is using money already budgeted for the existing phone system.
Lawmakers first got an inkling of troubles with the phone system last year, at an April 25, 2017, meeting of the Joint Ways and Means Subcommittee on General Government.
State officials and an IBM official casually told lawmakers at the April 25 hearing that implementation of the new system had been halted for four months because of widespread "dissatisfaction" from state agencies. That news shocked and dismayed committee members.
State Sen. Betsy Johnson (D-Scappoose) invoked OWIN and Cover Oregon, which she termed "catastrophic system failures."
"The worst thing in the [legislative] business is surprise," Johnson told a senior IBM representative and the state's chief information officer, Alex Pettit, that day. "This comes as an unwelcome surprise."
Pettit and the IBM official, Anthony Foster, told Johnson and her colleagues things would get better. They didn't.
The state of Oregon employs tens of thousands of people, many of whom use desk phones daily to serve the taxpaying public.
In 2015, state officials signed a contract with IBM to consolidate and modernize a mishmash of aging phone systems. The state faced a July 1, 2018, deadline to switch employees over from an existing contract with CenturyLink.
The new system IBM pitched wasn't cheap: The basic cost of designing it and supplying hardware was $39 million over 10 years. The Department of Administrative Services needed another $7 million for installation, bringing hard costs to $46 million. Annual maintenance will cost another $7 million or so a year.
Rather than conventional, hard-wired phones, the state bought a system that would rely on the internet, using what's called voice over internet protocol, or VOIP. (The state employs 50,000 people at more than 600 locations, but not all offices have the bandwidth for the new system.)
VOIP systems are supposed to be cheaper to use and maintain and more flexible than conventional phones. If employees change locations, for instance, they can take their numbers with them; their phones can be integrated with their computers, and adding new lines is far easier.
But from the beginning of installation in February 2016, documents show, state agencies experienced myriad problems, including dead phones, disappearing voicemail and poor sound quality.
Early last year, the state and IBM agreed on a "stop work" trouble-shooting period from Jan. 27 through May 17, 2017. At that point, about 20,000—or two-thirds—of the phones had been converted to the new system.
Despite a concentrated effort to bolster the system, it remained unstable, even as IBM raced to meet the July 1 deadline for getting all 30,000 lines up and running.
On Jan. 22 of this year, the state experienced outages that affected workers across the state. A backup "failover" solution IBM built also did not work. That was the tipping point for Pettit, the state's chief information officer.
"We apologize for the continued disruptions," Pettit wrote to all state agency directors Jan. 25, records show. "Put plainly, the outages and associated business impacts are unacceptable, and we have lost faith and confidence in the system."
The next day, Pettit's agency notified IBM that the contractor was in default and gave the company 30 days to find a solution. It's unclear what that solution will be.
"Projects of this size and magnitude are bound to experience some difficulties," says Craig, the DAS spokeswoman. "The contract we have in place provides remedy options for when obligations are not being met, which we have been using as necessary."
Gov. Kate Brown's spokesman Chris Pair says Brown is keeping a close watch on Project MUSIC.
"Many processes have been put in place to prevent another failure on the scale of Cover Oregon," Pair says. "Gov. Brown expects the chief information office to continue this work to ensure IBM resolves the recent outages and honor the terms of their contract. If IBM is unable to resolve these issues, the governor has authorized the state CIO to exercise any and all options available to restore reliable phone service."
In response to WW's written questions, IBM provided a statement but declined to comment further.
"IBM will continue to provide the services that the state has contracted us for, and we are working with the Department of Administrative Services to resolve the state's concerns," said company spokesman Clint Roswell. "We are committed to the continued success of this project."
IBM said the same thing to lawmakers nearly a year ago. Ways and Means Committee members warned the company—and Pettit—they wanted no further surprises.
But word of the default notice the state sent IBM has been slow to reach lawmakers.
"Unfortunately, this is news to me," replied state Rep. Greg Smith (R-Heppner), who co-chairs the Joint Ways and Means Subcommittee on General Government, when WW requested comment on the default notice. "I have not heard about this—and that's discouraging."