This problem is now resolved and closed.
17/06/2010 14:01 – We are aware of a problem with networking in our Derby PoP and have engineers looking at the problem.
14:12 – The problem has been resolved.
19:00 – The problem was resolved fully at 14:12 and further information will be posted here before end of business today.
19:30 – Further information on todays events can now be found below.
Information:
Further to the outage earlier today in our Derby PoP we have completed an investigation into the cause of the problem and what steps can be taken in future to prevent a recurrence.
At approximately 13:45 our intensive network monitoring alerted us to a minor DDoS against a specific service on our Derby network. Engineers immediately sprung into action identifying the target, source(s) and overall impact of the event.
Although the DDoS attack was only impacting on the specific service being attacked at ~14:00 a configuration change was made within the core network in an attempt to mitigate the attack. A mistake was made during application of the change causing it to be propagated across all core network devices and resulting in network outages for some customers.
Pipe Ten is committed to ensuring that lessons are learned in order to prevent such a problem happening ever again and as such we are undertaking a full review of our network change management policy and DDoS mitigation procedures.
We would like to take this opportunity to apologise again for this brief but critical outage. Customers wishing to discuss the problems in more detail should contact the Pipe Ten support team via the control panel trouble ticket system so an engineer may schedule a call-back.