Impact

Between 17:26 and 17:57 CET April 13, 2021 incoming and outgoing voice traffic through Telserv was not working. The Sipserv portal was also impacted and then subsequently disabled until 18:52 CET. All new inbound and outbound calls routed through Telserv failed during this outage window. it is difficult to assess the true customer impact - we’re working to pinpoint all customers affected and will update this page soon.

Prior to the outage we had a sustained level of approximately 1,200 active calls which dropped down to approximately 100 during the outage.

Root Cause

We’ve identified an unfortunate sequence of concurrent, incompatible events that led to a race condition in the platform’s main database. These events were actions performed in the portal by multiple customers around the same time:

• placing a backorder and adding items to cart
• running one or several reports (e.g. call quality report) that use the same table as the backordering process

This caused an abnormal load on our main database server, causing it to stop responding to further database queries.

This database issue affected the portal, partners API and voice infrastructure as all services are served by the same database. As a consequence no voice traffic was processed during this period: before a call is placed, we validate and retrieve routing configurations from the database to route the call accordingly. This is a pre-requirement for us to further process and place the call. As all database connections were getting either queued or refused, the voice processing flow would hold and retry continuously until successfully connecting to the database.

Mitigation

We are currently working on a number of follow-up actions to prevent this issue from happening again. These actions aim to add fault tolerance to report generation and segregate/ isolate voice infra database calls to prevent collateral effects:

• Move all reports to a new database server
• Create a read replica of the main database server and redirect all voice infra database traffic to use this server