May 18th 2022 - Delay in Console / NetFoundry API Services

 

[ISSUE RESOLVED] - The increase in API error rate and and provisioning delays have been resolved. The timeline of events is documented below:

[5-18-2022 11:30am UTC] - A spike in database activity caused some of the NetFoundry API services to become unresponsive. The process restarted on it's own around 8am, but problematic code was identified, which required a hotfix patch.

[5-18-2022 2:30pm UCT] - The hotfix deployment began, but a database migration in the deployment created a lock on the table. This caused an increase in 500 errors from our APIs.

[5-18-2022 3:00pm UTC] - The database lock was cleared, and API functionality restored. However, provisioning actions were backed up users were experiencing extended delays in provisioning activity.

[5-18-2022 4:10pm UCT] - Delays in provisioning were resolved, and API functionality returned to normal.

All of the issue cited above were localized to Netfoundry's provisioning API. At no time were NetFoundry's hosted fabric services impacted. Network functionality remained normal on customer networks.

 

 

[ISSUE IDENTIFIED] - Since 12.30 PM UTC Wednesday, 18th May 2022, we have been facing unexpected delays in our provisioning process ( www.nfconsole.io and our console APIs).  The impact of this is delay / temporary failures in creation or edit of components such as Networks, Edge routers, Endpoints, Services etc in our management and operations platform. By 2.30 PM UTC, we temporarily alleviated the issue. The delay has reduced but is still seen.  We are still working on the root cause and the permanent fix. We shall post an update once the issue has been fixed.

 

There is no impact to the run-time operations of customer networks.

 

Was this article helpful?
1 out of 1 found this helpful

Comments

0 comments

Please sign in to leave a comment.