2000 outages/events

web server, Dec 20 1:00 AM

The www.mv.com web server (and any aliases mapped to it) was down from about 1:00AM through 3:30AM due to a configuration error. This error had its effect at 1AM when the server is restarted nightly as part of some automatic processing. The error was corrected and the server brought back to life.

Shell server, Dec 8 3:30 AM

The shell server is temporarily down; it should be back up first thing in the morning. Cause: pilot error, as a fumbling admin (me, mem) working remotely inadvertantly put the system into single-user mode.
Status: Back up at 7AM.

October 20 12AM

Thursday 12AM: the shell server (iridium) will be down for a disk upgrade; downtime should be 2-3 hours. Once this upgrade is done, the web server will also be down briefly for addition of a new disk.

Mail/routing Oct 13

Late this afternoon and early this evening we experienced an overload of our mail server(s); this was caused by email requests coming into the server that started but never completed. A lot of mail comes into the mail server and each item is normally disposed of quickly; when mail conversations stall, the server can fill up with requests pretty quickly. So far it looks as though there is a problem receiving large packets that come in via the Globalcenter backbone; this has also affected other transfers (such as web requests) that come in via that path. We have temporarily turned off our connection to Destek (and thus Globalcenter), with the load being taken up by our other three main backbone connections.
Status Oct 14 00:30 : The problem was traced to a router configuration change within Globalcenter; this change has been backed out, and we've turned that circuit back up again.

News server Oct 6 1AM

The news server will be down at about 1AM for a (hopefully quick) reboot. Downtime should be a half hour or less.

Oct 1 12:00 AM news server

We'll be doing some work on the news server between midnight and 3AM (or so); the news server will be unavailable or have limited availability during that time.
Status: Back up at about 4:20AM. This is part of our gradual migration to new news hardware and updated software.

News server Sep 24 19:00

The news server is being shut down for a quick reboot; it will be unavailable for the next half hour or so.

Sep 22 1AM: AA box upgrade

At 1AM Friday Sep 22 the Assured Access remote access server will receive a software upgrade. This will affect users dialing into the 1-500- numbers as well as the "test" BNH -6388 numbers, but not users dialing into the standard BNH -6387 or other numbers. This outage should last less than an hour.
Status: Finished at 1:30.

Thu Sep 7, 2000 12:30am - 2:00am

Our news server was shut down while it was relocated to a new location within our building.

Aug 20 2000 21:10

We have an outage on an ethernet device in our NOC-- this is affecting some connectivity esp. to frame relay and leased line destinations. No ETR as yet.
Status: resolved as of 21:50

News server Aug 17, 2000

The news server was down for just over an hour starting at 12:50am for another test of new hardware. It was back up by 2:00am.

News server Aug 7, 2000

The news server will be down for about 30 minutes this afternoon for a test of new hardware. This will happen in the 1:30 to 2:30 timeframe.

Aug 1, 1AM: News server

The news server will be down Tuesday morning at 1AM while we install a new server engine. The disks will be moved to a new chassis and new motherboard. This should take around 2 hours.
This is number one of several incremental upgrades to the news server to be done over the next month or two.
Status: The memory shipped with the new motherboard was incompatible with that motherboard; although we have moved the drives to the new chassis, the server is still running on the old motherboard. When we receive the correct memory we will schedule another outage to complete the process. (Since most of the hard work is done, this should just be a matter of minutes.)

May 30, South NH Dial-up lines

Shortly after 8pm this evening we started receiving reports of problems reaching MV through several of the dial-up lines in the Nashua and surrounding regions. It appears at this time that the following exchanges are affected completely or intermittently:

Merrimack	420
Milford	732
Nashua	324
Pelham	751
Salem	824

The MV numbers in those exchanges ending in 6387 and 6388 are known to be affected. However, our 1-500-699-6387 number is still reachable from the affected regions so it may be used as an alternative.
Status 04:15AM: All the exchanges besides Nashua (324) appear to be back and working.
Status May 31 10:30AM: Nashua (324) now appears to be working.

May 25, News server

The news server is scheduled to be taken down sometime around 1am Thursday in order to replace a couple of its disks. The server is expected to remain down for somewhere between half an hour and an hour. During this time, ftp access into pop.mv.net will also be affected.

May 10, Mail server

This morning the mail server was unavailable for a short period: someone attempted to send an 880MB mail message and this filled up the temporary space on the server. (There is a 5MB mail message limit on the server, but this is only enforced after the size of the message is known). Email is not an appropriate vehicle for an 880MB data transfer.

April 26, 2000 11:40AM

The news server is experiencing disk problems and being worked on.
Status 16:00: There was a failure of the disk that has the history and other control information. We're doing a restore from tape onto a new disk, and once the data is restored we'll need to do a rebuild of the history file. This is likely to be an all-day operation.
Status April 27 5:00: The rebuild finished around 3AM but we discovered a bug in the rebuilding program. Although we ran the server for an hour or so, we decided that the bug had had such a negative effect that we needed to fix it and run the rebuild again. The rebuild is now ongoing and is expected to be done in the early afternoon.
Status April 27 12:45: News server is back up.

April 21 1AM

Rescheduled the disk upgrade in iron, see the entry below. We may have a way to use the disk that was shipped and will be trying that.
Status: upgrade has been completed.

April 20 1AM

One of our internal servers providing nameservice and authentication (iron) will be taken down at this time for a disk upgrade. Since this is a redundant server, most users should not notice the downtime.
Status: This operation was not successful due to incompatible disk drives received from the vendor. We will reschedule it when it's been corrected.

March 23 2000 - Shell server

The shell server experienced a disk failure just before noon. We have installed a replacement disk and are working on restoring data from the most recent backup tape.
Status 15:30 : We're able to restore the data (with the exception of a few files) directly from the old failed disk. This takes some care and patience, but will result in a more complete restoral than from tape (we did have a tape backup that is as recent as 10:00 this morning, but even restoring from that would mean the loss of 2-3 hours worth of shell mail, thus we prefer recovering from the failed disk even though it is more tedious).
Status 16:35 : Shell server is back up.

News Server March 20

The news server experienced a disk failure; it will be down for at least some of this morning while this problem is repaired.
Status 10:00 : No hardware problem could be immediately identified. The server is back up and running in the same configuration, we will continue to watch it.

News Server March 15 1AM

The news server will be down starting at 1AM on Wednesday morning March 15 (that's the night of Tuesday/Wednesday the 14th/15th). The server needs to be moved to another location for a few weeks while the area that it is in is being worked on. The outage should last less than an hour.

Friday Feb 26, 1-500-699-6387 number in Nashua area

The new statewide number (1-500-699-6387) is out of service in the southern New Hampshire area (the Nashua service area for this number). The problem started at around 4:30 this morning, and has been given to Bell Atlantic for resolution.
Status: Back up as of about 1:55pm, was found to have been busied at the Bell Atlantic switch.

Sat Feb 19 1AM, Assured Access server

Overnight Fri/Sat we will be doing a software upgrade on the Assured Access remote access server. During this time, the "test" numbers (Brooks xxx-6388 and Bell Atlantic 1-500-699-6387) may be unavailable for a short period. If the upgrade goes well, the the process should only take a half hour or less.
Note that last time we tried an equivalent upgrade, the new software had a problem and we had to back it out without the server ever fully coming back on line. This is a newer release and we hope that will not happen again, but if it does the downtime could be longer than expected.
Status 2:10 Upgrade successful with only a small amount of downtime. We will continue to watch for any side-effects.

Feb 11 2000 - internal routing outage

From about midnight through 1:30AM we experienced a series of routing problems within the MV network making some servers and connections unavailable.

Feb 8/9 2000 - various nationwide Internet problems

As you've probably read, there have been a number of distributed denial of service attacks against several major Internet destinations in the last few days. Along with this (and probably related) there have been several outages or slowdowns in various major Internet backbones. For example, during the evening on Feb 8, the Alternet (UUnet) backbone was nearly unusable, and in fact for a while we had to disconnect our UUnet circuit and let our other backbone connections take up the slack (the recent Cable and Wireless addition helped.) If you are seeing problems getting to certain places on the Internet, be aware that there have been these sorts of problems.

Brooks 420- all circuits busy. Feb 2, 2000

Customers in Merrimack have reported receiving an All Circuits Busy message when dialing our 420-6387 number. The problem has been reported to the phone company. Customers are advised to try a different number until the situation has been resolved. Available numbers.
Status: Customers report that on Feb 5, 2000 the problem has cleared up.

BNH dialines Jan 13 2000

Not necessarily an outage: In response to some odd behaviour of our PM3 remote access servers over the last few weeks, we have downgraded the operating system in the BNH remote access servers from 3.9b22 to 3.8.2c4 . We had been running 3.9b22, which is an open beta release, since about October 8. 3.8.2c4 is a non-beta of an earlier release- this code is actually more recent than 3.9b22 and contains the same modem code (or later code) as 3.9b22. This downgrade occured between 7:30 and 8:00 on January 13.

Shell server Jan 12 2000

The shell server will be down briefly at around 1AM Wednesday morning, January 12, in order to add some disk capacity (adding in a disk that was in the old shell server). The outage should be signficantly less than an hour.
Status: Back up as of about 1:40.

Litchfield Saturday January 8 2000

There will be some power work at the Litchfield site as, apparently, a new pole is being put in. There is no ETA on the start of the outage, and we are told it will last anywhere from 30 minutes to half the day.

We have very little service remaining in Litchfield, most of it having been moved to our Manchester NOC. Primarily, this outage will affect Frame Relay customers, who are connected through Litchfield (note: this service is due to be relocated from Litchfield to Manchester later this month).

Unfortunately we received extremely short notification of this event; when we got the notice on Friday evening we installed some extra UPS power at the site- this should help to cover a short outage but if it lasts for hours, expect the power to go down. (The process of installing the UPS equipment may have caused some very short interruptions for Frame Relay customers.)

Status: There was an outage that lasted from about 8:15 through 10:45.