Account services

System crash Friday 22 September 2017

Released on 2017-09-24 by: Søren Roug

On Friday 22 September at 11:30 CET IT operations began the work to make an upgrade of a service management tool on the test servers. This upgrade causes the server to stop all applications it is running, upgrade and then start the applications again. The upgrade was considered routine.

By mistake the command to upgrade was applied to all servers - not just the test servers. This caused widespread shutdowns and it overwhelmed the system administration tools. They were also being upgraded and crashed.

The incident lasted until around 20:30.