Delayed Connectivity
Incident Report for Cytracom
Postmortem

Summary:

Last week Cytracom experienced a throttling of our bandwidth by our hosting provider. While this did not result in a service outage, customers did experience delayed call setup when placing and receiving calls. This was an unacceptable service degradation that was unintentionally caused by our cloud hosting provider placing a traffic limiter on our connection. This limiter was applied to our connection inadvertently roughly two years ago and was done so without our knowledge or consent.

What we’ve done so this will not happen again:

We worked with our provider to identify and correct the issue. Our rapid growth led to this problem -- more customers requires more bandwidth. The traffic limiter that had been placed on our connection had been out of sight and out of mind and had not been configured to keep up with the demand. Going forward, we no longer have bandwidth threshold limits, so we should never have this issue again.

We realize that Cytracom is an important part of your business, and apologize for the impact that this degradation had to your operations. We take great pride in providing the best possible service quality to our customers. Occasionally, we run into problems as detailed above. These incidents further drive us to continually improve the quality of our own internal operations and ensure that we are living up to the trust you have placed in us. We are working diligently to provide you with a stable and pleasant experience and thank you for your continued support!

Posted Mar 28, 2017 - 23:04 CDT

Resolved
The fix put in place during the maintenance window was successful. We will continue to monitor the network and provide a full post-mortem.
Posted Mar 24, 2017 - 01:05 CDT
Update
Customers should no longer be experiencing connection delays at this time. We will be pushing a permanent fix this evening during an emergency maintenance window from 11pm - 1am CST. We expect there to be brief interruptions to service during this window and will update this page once complete. We apologize for any inconvenience and will provide a full post-mortem for this incident.
Posted Mar 23, 2017 - 20:07 CDT
Identified
Customers are experiencing delayed connectivity when accessing Cytracom services. We believe we have identified the problem and are working toward resolution.
Posted Mar 23, 2017 - 17:58 CDT