[KC-Justin]

22:20 Tuesday March 16th.

The London Internet Exchange (LINX) suffered a major failure, resulting in BGP sessions flapping and causing intermittent packet loss among ISP's present at the exchange.

This severely impacted in/out traffic from the KillerCreation Network. As a result, at 23.10 tonight we shutdown all LINX peering sessions and will continue to monitor the situation.

[KC-Justin]

Update from LINX

Quote

The LINX 195.66.224.0/23 peering LAN (Brocade LAN) was experiencing MRP
flapping. This has severely impacted traffic.

I have shutdown the redundant links in the two MRP rings wich were
flapping. This has stabilised things.

At the moment I am in the process of tracking down the root cuase of the
MRP flaps.

I will provide a further update within the next 45 minutes

-- London Internet Exchange Ltd - http://www.linx.net

[KC-Justin]

Update 2

Quote

Things didn't stabilise as much as I initially thought. This outage is
ongoing.

The instabilities have caused the Quagga Routeing Daemon on LINX Route
Server rs3.linx.net to fail and al BGP sessions are currently down. This
will be left down until the peering LAN has been stabilised.

We're still attempting to track down the root cause of the problem.

I will provide another update within the next 45 minutes

--
London Internet Exchange Ltd - http://www.linx.net

[KC-Justin]

Final update for the evening. Another update will follow in the morning.

Quote

We are continuing to work on this problem, but I have no significant
updates at the moment.

We are still trying to isolate the root cause of this problem.

There is a LINX engineer on-site in Docklands should we need to work on
any hardware issues.

I will provide another update within one hour.

-- London Internet Exchange Ltd - http://www.linx.net

[KC-Justin]

Quote

The LINX 195.66.224.0/23 peering LAN (Brocade LAN) is bow stable and
operating normally.

We have restored the above Switch2 to Switch9 link and the redundant
links that were disabled at the start of the outage.

We are still analysing all the data to determine the exact cause of the
problem.

Although the exact cause has yet to be determined, LINX does appear stable. We have re-enabled BGP sessions and will continue to monitor traffic.

Our peering sessions at LINX cover approx 40% of our total traffic on the KillerCreation Network. When issues like this occur, traffic is re-routed via alternate paths so no destination is unreachable.

[Sgt.Trojan]

That was one crazy night on BC2. Some people suffered terrible lag but most of them seemed to be ADSL customers. I'm on cable and had very little problem.

It just proves how vulnerable we are and how our little online worlds come crashing down due to one piece of hardware. 8)

[JTRipper]

Sgt.Trojan, on 17 March 2010 - 10:34 AM, said:

That was one crazy night on BC2. Some people suffered terrible lag but most of them seemed to be ADSL customers. I'm on cable and had very little problem.

It just proves how vulnerable we are and how our little online worlds come crashing down due to one piece of hardware. 8)

yeh we had the same problem

ADSL members were complainging of lag, us cable lads were fine