Provisioning Services Delays

During last days and mostly on 28 November, 2025, some customers experienced *delays of up to one hour in provisioning actions that normally complete within minutes such as new purchases, subscription changes, and other service requests.

Our platform performs a large batch of automated subscription renewals. During the most recent run, the number of renewals was significantly higher than usual. At the same time, customers were submitting new purchases and change requests that also required provisioning.

Provisioning operations are handled asynchronously and rely on an internal queueing mechanism (“outbox queue”) that processes various background tasks, including external pricing.

The increased volume of subscription renewals triggered an unexpected bottleneck in this queue, slowing down the processing of real-time provisioning actions.

The key contributing factors were:

• High volume of subscription renewals especially on the 28th of the month.

• Asynchronous provisioning architecture, which relies on the outbox queue to complete external pricing calls for both renewals and new orders.

• Queue prioritization behavior that caused renewal-related tasks to temporarily dominate throughput, slowing down processing of other provisioning actions.

This combination resulted in slower processing times and delayed provisioning for customer-initiated actions.

The issue was identified on 28 November, 2025 and a hotfix for Queue prioritization behavior was immediate deployed helping on restore normal provisioning times.

Impact

• Delayed provisioning for new purchases, subscription changes, and other customer-initiated actions.

• Delays lasted up to approximately 1 hour depending on the request type and time submitted.

• No data integrity issues occurred, and no actions failed; all were eventually completed successfully.

What We’re Doing Next

To prevent this from happening again, we are implementing the following permanent improvements:

Queue Prioritization Enhancements

• Adjusting priority rules to ensure customer-initiated provisioning actions maintain high priority even during peak system events like monthly renewals.

Monitoring Upgrades

• Enhancing predictive monitoring to identify potential bottlenecks before they impact customers.
• Increasing alerting sensitivity for queue buildup.

Current Status

The system is fully operational, and provisioning times have returned to normal. We continue to monitor performance closely.

We sincerely apologize for the inconvenience this caused. Ensuring a reliable and seamless experience for our customers is our top priority, and incidents like this help us continually improve the resilience of our platform.