BLF and Park Keys: Degraded Performance
Incident Report for Cytracom
Postmortem

BLF Outage

Between Thursday, October 22nd at 5:13 AM CST and Friday, October 23rd at 6:34 PM CST, Cytracom BLF functionality was impacted while undergoing routine maintenance on a core component interacting with the BLF service. The result of this procedural failure was continual "soft" failures within the BLF service. The normal recovery reactions resulted in an overload of traffic and consumption of capacity.

Corrective measures have been implemented in several areas including but not limited to: maintenance procedures, rate-limiting for BLF nodes and capacity planning for the addition of future nodes.

Posted Oct 31, 2020 - 08:39 CDT

Resolved
This incident has been resolved.
Posted Oct 23, 2020 - 22:35 CDT
Monitoring
We have mitigated the issue that affected BLF and park status key performance, and we will continue monitoring the situation.
Posted Oct 23, 2020 - 15:44 CDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 23, 2020 - 11:02 CDT
Investigating
We are currently investigating an issue that is causing degraded performance of BLF and park status keys. We will continue to post updates here as soon as they come available.
Posted Oct 23, 2020 - 09:38 CDT
This incident affected: Voice Services.