danbricklin.com/log

ISP Outage Explanation
This is an email we received to explain outages in our connections to the Internet. Names, etc., have been changed to XXXX, etc. While they refer to "65 IP customers" each customer may be a company like us with many users. It's like saying "65 blocks in the city" not saying if they have single houses or apartment houses.

From:      WWW, YYYY[SMTP:YYYY_WWW@XXXX.com]
Sent:      Wednesday, November 03, 1999 4:48 PM
To:      'zzzz@trellix.com'
Subject:      POST MORTEM

Dear Customer:

This is a Post Mortem to inform you of the outage for RRRRR.


POST MORTEM
RRRRR  OUTAGE


SUMMARY OF OUTAGE:
65 IP customers down due to bad hardware on RRRRR router

Customers off of PPPPP were down on 2 separate occasions:
October 29, 1999: 8:15am - 11:41am (3hours and 31 minutes)
November 1, 1999: 3:47am - 11:53am (8 hours and 7 minutes)

Customers off of QQQQQ were bouncing constantly starting on October 29, 1999 and continuing through October 31, 1999.


DETAILED ACCOUNT OF TROUBLESHOOTING:

On Friday, October 29, 1999 beginning at 8:40AM EST we began experiencing outages in New York.  The problem originally appeared to be due to a circuit outage but after several iterations of troubleshooting we discovered the problem to be with the XXXX router - RRRRR.

Several organizations within XXXX were pulled together to attack the issue
- Field Support, VVVVV, TTTTTT - as well as
engineers from our first, second, and third level support.  SSSS, the
manufacturer of our RRRRR router has a case open to investigate
problems on this router also.  Several pieces of hardware were replaced and
several hours of testing between 2 different POP sites and 2 operations
centers were conducted over the course of 3 days. Service was temporarily
restored at different intervals during the troubleshooting process, however,
multiple problems were found and had to be corrected.

The last outage occurred and was restored on Monday, November 1st .  We
believe that all outstanding problems have been addressed and our second and
third level engineers are continuing to closely monitor the router.  The
case with SSSS is still open and under investigation.  

We are sorry for any inconvenience we have caused.  If you have any further questions or concerns, feel free to contact your sales representative or contact us via e-mail at support@XXXX.net.


XXXX IP Operations

© Copyright 1999-2017 by Daniel Bricklin
All Rights Reserved.