Dear all
The failing switch has been isolated for more than an hour now and we have seen all services have returned to normal.
We are therefore closing the incident and will conduct a post mortem this week to see what failed and what we can do improve the situation.
Sorry for the problems this has caused you.
Best regards
Jens-Christian
--
Jens-Christian Fischer, Head Community SOC
SWITCH
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
https://switch.chhttps://swit.ch/linkedinhttps://swit.ch/twitter
Dear all
We have found a problem with one of our network switches that blackholed a lot of internal traffic. This caused problems in VMs accessing our storage cluster which blocked those VMs.
After draining the switch, traffic is flowing normally again and access to the storage cluster has been restored. We are seeing affected services coming back online.
We are now monitoring the situation closely.
Best regards
Jens-Christian
--
Jens-Christian Fischer, Head Community SOC
SWITCH
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
https://switch.chhttps://swit.ch/linkedinhttps://swit.ch/twitter
Dear all
Since 6.6.2022, 12:53 we are experiencing various problems on SWITCHengines. This has led to several of your services failing as well.
We have no engineers on the case and have restarted one of the failed services (OpenStack Glance - the image service). There are still problems with various VMs in hung states. We are investigating.
Next update: 16:00 or check our status page : https://status.switch.ch/ <https://status.switch.ch/>
Best regards
Jens-Christian Fischer
--
Jens-Christian Fischer, Head Community SOC
SWITCH
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
https://switch.chhttps://swit.ch/linkedinhttps://swit.ch/twitter
Dear SWITCHengines Users,
As part of routine preventative maintenance, our next OpenStack version Train, is ready to deploy.
As with previous upgrades, to reduce impact we upgrade individual components on a rolling basis with those that have a higher risk taking place in the maintenance windows.
The upgrade calendar for Train is as follows:
18.1.2022 - Keystone Authentication Service (Global Change).
Lausanne will be upgraded between 11th and 27th January
Zürich will be upgraded between 13th and 27th January
During this time, access to VMs is not expected to be impacted, and control plane operations (creating, deleting or changing VMs) may experience a few moments interruption during the maintenance windows of Mondays 21:00-Tuesdays 09:00 and Thursdays 21:00-Fridays 09:00.
Between 2020 and now, we have run an aggressive upgrade schedule for OpenStack to improve stability and maintainability with multiple upgrades per year. After Train we will be taking a pause for 6-9 months on OpenStack upgrades to synch in various dependent systems.
If you have any questions about the upgrade, please let us know at engines-support(a)switch.ch<mailto:engines-support@switch.ch>
Happy New Year,
Ann
--
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengines users,
SWITCH offices will be closed from Friday December 24th to Friday December 31st inclusive. During this period we will run a reduced support service covering incidents and serious/time sensitive issues. Regular service requests will be processed when we return fully in January.
The SWITCHengines team would like to thank you for using SWITCHengines during 2021, for your active and constructive engagement with us during moments both good and bad and we wish you a peaceful holiday season.
All the best,
Ann & the SWITCHengines crew
🎄🎄🎄🎄🎄
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengines users,
Overnight we experienced some instability of the object storage gateways in the Ceph object storage (v 1). This resulted in data being unavailable, and also impacted some dependent services such as Gitlab. We stabilised the situation at 08:45 and are investigating.
Apologies for the difficulties caused.
Regards,
Ann
--
Ann Harding, Team Lead, Infrastructure & Platform as a Service,
SWITCH, Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 253 98 14, ann.harding(a)switch.ch<mailto:ann.harding@switch.ch>
Working for a better digital world - www.switch.ch<http://www.switch.ch>
Dear SWITCHengines users,
Since 16:00 we are observing IPv6 connectivity issues for a range of SWITCHengines VMs. We are investigating and working to restore access.
If you have any further questions, please contact us at engines-support(a)switch.ch<mailto:engines-support@switch.ch>
All the best,
Ann Harding
Dear SWITCHengines users,
The SWITCHengines Horizon dashboard and Quickstart simplified web interface are currently unavailable. VMs are running normally but it is not possible to make changes to VMs or to provision new projects or VMs via the web. API/command line access is still available. We are investigating.
All the best,
Ann Harding
Dear SWITCHengines users,
We have observed an issue with provisioning SSH keys when starting new VMs or rebuilding existing VMs on the main virtual router in Zürich. We have identified the cause and verified and have some workarounds but have still to define a persistent fix. We advise that you do not restart machines if possible until the issue is fully fixed.
If you have any questions, please contact us at engines-support(a)switch.ch<mailto:engines-support@switch.ch>
All the best,
Ann Harding
Dear SWITCHengines users,
On Wednesday, August 4th, between 6 a.m. and 10 a.m., essential electrical work will be taking place at the site of our Zurich region. As all systems have dual-homed power, this should not be service affecting.
However, the electrical work involves cutting regular power to the entire data center, causing a simultaneous failover of all hardware to battery power. Since this comes with an elevated risk of hardware failures, we announce this maintenance window as a precaution.
Best regards,
Michael Helminger
--
SWITCH
Michael Helminger | Infrastructure and Platform as Service
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15 | direct +41 44 268 15 28
michael.helminger(a)switch.ch | https://www.switch.ch
