The outage experienced by some customers on the morning of Saturday, May 26th has been analyzed and corrective actions put in place to mitigate the risk of this failure in the future. The following is a basic overview of the analysis and actions:
Systems Affected:
The systems supporting voice services and used in making inbound and outbound phone calls.
Impact:
The majority of customers were unable to complete inbound or outbound phone calls.
Problem:
Transient state tracking on shared resource was not responding even though all health checks were normal.
Recovery:
Restored transient state tracking from the shared resource to distributed dedicated resources.
Corrective Action:
Implemented additional monitoring dedicated to each distributed resource.
Posted Jun 01, 2018 - 16:40 CDT
Resolved
This incident has been resolved.
Posted May 26, 2018 - 12:46 CDT
Monitoring
Full resolution of the underlying call completion issue is in place. Cytracom Engineering will continue to monitor.
Posted May 26, 2018 - 11:40 CDT
Investigating
Cytracom Engineering has identified an issue impairing voice services for many of our customers. We are investigating the root cause and working to a full resolution. As updates become available, they will be posted here.
Posted May 26, 2018 - 10:12 CDT
This incident affected: Voice Services (Call Completion).