Delivery issue to Netherlands operators last night
Incident
Report for Bird
Resolved
Last night at 6:00pm CET the primary connection for the T-Mobile network in NL had to be adjusted for unexpected maintenance reasons on the network carrier’s side.
Our failover-system switched to the redundant, direct connection in place, which unfortunately showed delays caused by high volume of messages being sent at once. Due to an internal upgrade of our platform’s route performance monitoring system, the initial designated monitor did not recognise the delays and hence didn’t trigger the next failover correctly. Because of this, the issue that occurred was not corrected automatically and in the usual timely manner so that no impact would’ve been experienced.
We have taken further measures by reassessing the monitoring in place as well as adjusting the internal notification scheme to increase visibility for future occurrences. The issue had been resolved automatically on the connection’s side as well as on our side by manually interfering in the failover system.
Our failover-system switched to the redundant, direct connection in place, which unfortunately showed delays caused by high volume of messages being sent at once. Due to an internal upgrade of our platform’s route performance monitoring system, the initial designated monitor did not recognise the delays and hence didn’t trigger the next failover correctly. Because of this, the issue that occurred was not corrected automatically and in the usual timely manner so that no impact would’ve been experienced.
We have taken further measures by reassessing the monitoring in place as well as adjusting the internal notification scheme to increase visibility for future occurrences. The issue had been resolved automatically on the connection’s side as well as on our side by manually interfering in the failover system.
Investigating
We've experienced an issue with SMS delivery that impacted sending to some networks in the Netherlands yesterday from 6:00pm CET (31/01/18) to 00:40am CET (01/02/18). Whilst this has since resolved itself we're currently still investigating what caused this and will update this notification shortly. Our apologies for the inconvenience.