Dublin - Cooling issues - 06/02/2008
Closed.13:09 - We are pro-actively powering down elements of the shared hosting cluster due to cooling related issues
13:11 -
Cooling in the Data Centre is being affected due to main power board supply failure to the chiller units. This is causing the Data Centre to heat up and as a precautionary measure we are shutting down all non-critical servers to protect the hardware.
The cause has been identified and a fix is being implemented immediately. We expect the work to take approximately 20 minutes, and potentially a further 20 minutes to return the Data Centre to normal temperature - at which point all equipment will be safely powered back up.
13:35 - The chillers are back on-line and temparture is returning to normal in the Data Centre. We expect normal service to resume in 10-15 minutes.
14:00 - Most services should now be restored, failed services are being worked on. Please expect small delays in mail delivery as an incoming backlog is processed. No email should of been lost (when sent via a properly configured mail server).
15.10 - Further explanation of todays earlier issues: There was an electrical failure within one of the local fan coil units (local cooling unit) on the Data Centre floor, which caused the fuses for the board supplying the fan coil units to trip, thus protecting the infrastructure. The issue was localised to the effected fan coil unit, the fuses replaced, and the supply brought back online within 20 minutes. Without the cooling units maintaining the environment in the Data Centre, the temperature increased significantly, forcing us to perform a controlled shut down on equipment, which commenced 10 minutes into the issue. This was to protect the hardware/data integrity. There are very few remaining servers/services offline and email continues to process the backlog of messages.