21th of April - 20:45 UTC
Completion of Non-Production Environment Migration
The migration of all non-production environments to our new, stable cluster is now complete. This concludes our recent series of infrastructure enhancements.
Your non-production environments are hosted on a more robust and reliable platform, ensuring better performance and improved stability. As you begin using these environments, our support team is ready to assist with any questions or needs you may have.
We appreciate your patience throughout this migration process and are here to support any further needs or feedback.
Thank you for your continued partnership.
21th of April - 18:45 UTC
We continue to start the non-production environments that are down in batches and monitor the infrastructure as this is being done.
A next update to the KBA will be posted at 21:00 UTC at the latest.
21th of April - 17:35 UTC
We have completed the successful migration of all production environments to our new cluster. The maintenance window is now closed. Please note, the migration of non-production environments is ongoing and will be completed shortly.
Thank you for your patience during this transition.
A next update to the KBA will be posted at 18:30 UTC at the latest.
21th of April - 14:50 UTC
We would like to inform you that the ongoing maintenance activities require an extension of the planned window by an additional 2 hours. This extension is essential to migrate all customer instances to a new, stable cluster, ensuring the reliability and robustness of our services.
We understand the critical nature of these services to your business and apologize for any inconvenience that this extension may cause. Our technical team is fully committed to executing this migration efficiently and restoring full-service functionality as our top priority.
A next update to the KBA will be posted at the end of the maintenance at 17:00 UTC at the latest.
21th of April - 13:00 UTC
During our ongoing maintenance activities, we have identified that we need to ensure the stability and performance of our services further. To ensure the highest standards of service quality, we have decided to extend the current maintenance window by an additional 2 hours.
This extension will allow us to migrate all customer workloads to a newly stabilized cluster, ensuring a robust and reliable environment for your operations. We anticipate that there will be an outage during this migration process. We are taking all necessary steps to minimize the downtime and its impact on your services.
We understand the importance of our services to your business and apologize for any inconvenience this extension may cause. Our team is working diligently to complete the process as quickly and smoothly as possible.
A next update to the KBA will be posted at the end of the maintenance at 15:00 UTC at the latest.
21th of April - 11:30 UTC
We are halfway through our scheduled 4-hour maintenance window. Our team is actively engaged in troubleshooting and stabilizing our Docker cluster to enhance our services' performance and reliability.
While you may experience intermittent availability with our cloud portal during this period, please rest assured that all processes continue to run and are being closely monitored.
We appreciate your patience and understanding as we work to complete the maintenance as swiftly and efficiently as possible. We are committed to restoring full service functionality and will provide another update as we approach the conclusion of our maintenance activities.
21th of April - 09:00 UTC
Dear Customers,
the announced emergency maintenance has started. We will update at the end of the maintenance at 13:00 UTC at the latest.
20th of April - 18:00 UTC
Dear Customers,
We have stabilized the cluster traffic and the cloud portal by reducing capacity to the stable nodes. To continue our stabilization efforts and increase resources and capacity of the cluster, we will initiate an emergency maintenance window at 9 AM UTC tomorrow. The maintenance is targeted to last for approximately 4 hours, with a possibility of extending up to 6 hours if necessary.
While we strive to minimize disruptions, this maintenance is essential for enhancing service reliability and performance. We appreciate your understanding and patience during this period.
A next update to the KBA will be posted at the end of the maintenance at 13:00 UTC at the latest. We'll communicate whether an extension is needed.
20th of April - 16:00 UTC
Dear Customers,
All tenant systems are now showing healthy statuses. However, we are still experiencing intermittent availability with our cloud portal. We continue to work on fully stabilizing the cluster and are actively troubleshooting to identify the root cause to ensure consistent and reliable service.
While all functionalities are accessible, our team remains committed to investigation and vigilant monitoring to prevent future disruptions.
A next update to the KBA will be posted at 18:00 UTC at the latest.
20th of April - 14:00 UTC
Dear Customers,
We have detected unhealthy nodes within our production cluster in Dublin Region. This is causing intermittent disruption of service for all customers in that region. We are actively troubleshooting the issues and replacing the unhealthy nodes causing the issues.
A next update to the KBA will be posted at 16:00 UTC at the latest.
20th of April - 12:00 UTC
Dear Customers,
Our Cloud Infrastructure team is still working on rerouting the required services to a new node in the infrastructure and checking if the new node is working fine and stable.
A next update to the KBA will be posted at 14:00 UTC at the latest.
20th of April - 10:30 UTC
Dear Customers,
Our Cloud Infrastructure team has taken the affected infrastructure component offline and continues to work on rerouting the required services to other nodes.
Unfortunately you may still experience connection issues and delays when connecting to the dashboard or your environments, and also agents connecting to your environment may briefly disconnect and will try to reconnect.
A next update to the KBA will be posted at 12:00 UTC at the latest.
20th of April - 09:30 UTC
Dear Customers,
Our Cloud Infrastructure team has identified the affected infrastructure component and continues to work on addressing the problem.
Symptoms of the issue are intermittent connection issues to environments and intermittent connection issues to the Cloud Dashboard.
A next update to the KBA will be posted at 10:30 UTC at the latest.
20th of April - 08:00 UTC
Dear Customers,
We have created this knowledge base article to keep you apprised of the current developments regarding today's outage in the Dublin Region.
Our monitoring system identified the ongoing issue, and our Engineering team was notified.
We are actively working on the recovery of the environment(s) that is(are) affected.
The current focus of the team is to bring affected infrastructure back online, once that is completed and the environments were verified, the team will work on providing a more detailed RCA.
We will keep posting regular updates in this KBA.
Comments
12 comments
KBA updated
KBA updated
KBA updated
KBA updated
KBA Updated
KBA updated
KBA updated
KBA updated
KBA updated
KBA updated
KBA Updated
When can we expect an RCA on this?
Please sign in to leave a comment.