UK – Sheffield – Network Issue – 08/10/2009
Closed.
9:45 – We are aware of a network issue affecting our Sheffield POP which is being investigated by an engineer.
10:00 – The source of the issue with the upstream provider has been identified and they are currently working on fixing the problem.
10:20 – Connectiity has been restored and normal service should now have resumed. If you are still having any issues, please contact support using the usual method.
Further information:
This problem was caused by a hitless OS upgrade on a core router scheduled at 8am. This upgrade failed to work correctly as it is designed to upgrade seamlessly and not interrupt service. Initially the upgrade appeared to be fully functional, then at 09:30 the router began routing loops and causing a complete loss of transit.
What actually was happening should be impossible, this pointed towards to the probability of a software bug within the core router (Foundry) .Due to the style of the fault, more time than expected was taken to trace the route cause.
The issue was finally resolved by rebooting the core router at fault.
The issues has been raised this with the vendor to establish why the hitless reload process did not work and future maintence of this type has been rescheduled for a more appropriate time.
