Dear SWITCHengines users,
As part of our commitment to improve the stability of SWITCHengines and modernise the available features, we have been preparing upgrades to OpenStack to be carried out over 2020. Typically an OpenStack upgrade requires between 7-10 individual components to be upgraded. To minimise service downtime we planned and tested a rolling set of updates, component by component rather than a single larger period of unavailability to upgrade the full system in one go. While many of the components can be upgraded with low risk in-service, three require brief interruptions to service and are scheduled in maintenance windows. The Keystone component for authentication has already been successfully upgraded to OpenStack Pike.
The next scheduled components are Nova, which enables provisioning and de-provisioning of virtual machines, and network agents which manage the virtual networking. The strategy will be to upgrade both components in one region, and then both in a second region.
The proposed schedule, all within maintenance windows is as follows:
Thursday 13th August 21:00-23:00 OpenStack Nova Zürich - provisioning/deprovisioning of VMs
Expected impact: A few moments of unavailability of functionality to provision, deprovision or change VMs. Running VMs are NOT impacted.
Tuesday18th August 07:00-09:00 OpenStack Network Agents Zürich - virtual networking functions
Expected impact: Virtual networking functionality will be unavailable for a few moments per network agent during this maintenance window. This may impact connectivity to individual VMs on a rolling basis.
Thursday 20th August 21:00-23:00 OpenStack Nova Lausanne - - provisioning/deprovisioning of VMs
Expected impact: A few moments of unavailability of functionality to provision, deprovision or change VMs. Running VMs are NOT impacted.
Tuesday 25th August 07:00-09:00 OpenStack Network Agents Lausanne - virtual networking functions
Expected impact: Virtual networking functionality will be unavailable in for a few moments per network agent during this maintenance window. This may impact connectivity to individual VMs on a rolling basis.
If you have any questions about the planned upgrades or have some concerns about concrete proposed dates, please let us know at engines-support(a)switch.ch<mailto:engines-support@switch.ch>
All the best,
Ann
--
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengines users,
In order to better manage resources in our infrastructure, and to help you control your costs, we are changing the way shutoff Virtual Machines (VM) are being handled. We will be actively shelving VMs which have been shutoff for more than one month to make more infrastructure available. Shelving a VM means taking a snapshot and putting it in storage. No data is lost and you will only pay for storage and IPs while it is shelved.
If you use scripts or other forms of orchestration to launch VMs, you may need to update your code. After a VM has been shelved, it can’t be started with “openstack server start [server]”. To unshelve your VM use Quickstart or “openstack server unshelve [server]”. If you have problems unshelving your VM or further questions please get in touch with us via engines-support(a)switch.ch<mailto:engines-support@switch.ch>.
We will start implementing this initially manually on a subset of VMs before automating.
Kind regards,
Ann Harding
--
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengines users,
As part of our commitment to improve the stability of SWITCHengines and modernise the available features, we have been preparing upgrades to OpenStack to be carried out over 2020. To minimise service downtime we planned and tested a rolling set of updates, component by component rather than a single larger period of unavailability to upgrade the full system in one go. As the critical mass of components for the Pike release have been successfully completed, it is possible to schedule the Queens upgrade.
Maintenance to SWITCHengines to upgrade the authentication component Keystone to OpenStack Queens is planned to take place on the evening of Thursday 27th August between 21:00-23:00
What can you expect for impact?
• This change is global across both regions.
• Significant refactoring took place that now allows us to upgrade with only a few moments downtime within the longer scheduled maintenance window.
• No immediate changes will be necessary on the user side to resume using the service
Please let us know if you have any questions at engines-support(a)switch.ch<mailto:engines-support@switch.ch>
Regards,
Ann
--
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengiens users
The problem that caused the Ceph cluster to become unavailable has been solved. We are continuing to monitor the situation but will close the incident.
A technical postmortem will be published next week
Thanks for your understanding
Jens-Christian Fischer
--
SWITCH
Jens-Christian Fischer, Teamlead Managed Applications & Services
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fischer(a)switch.ch<mailto:jens-christian.fischer@switch.ch>
http://www.switch.ch
Dear SWITCHengines user
The network problems caused problems with the Ceph cluster which stalled all IO operations. The cluster is back and working again and VMs are working again. If you are still experiencing problems, please contact us via engines-support(a)switch.ch<mailto:engines-support@switch.ch>
The situation is stable now and we continue to monitor it closely.
We are leaving the incident open for now, and will close it at 16:00 if everything continues to work.
Thanks for your understanding
Jens-Christian Fischer
--
SWITCH
Jens-Christian Fischer, Teamlead Managed Applications & Services
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fischer(a)switch.ch<mailto:jens-christian.fischer@switch.ch>
http://www.switch.ch
Dear SWITCHengines user
We are experiencing a network problem in the ZH region. All VM / storage operations are currently affected.
We are investigating the problem and are working on fixing it.
Next update: 14:00
Best regards
Jens-Christian Fischer
--
SWITCH
Jens-Christian Fischer, Teamlead Managed Applications & Services
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fischer(a)switch.ch<mailto:jens-christian.fischer@switch.ch>
http://www.switch.ch
