2003 outages/events

Dec 13: web server

The web server will be undergoing some maintenance during the afternoon Saturday to replace a disk drive. We expect a brief downtime starting around 1pm, and another brief downtime later in the day. (i.e., two reboots will be required.)
Status: Postponed to a later date due to an equipment issue.

November 3: 1-500 number going away

As we've pointed out in the newsletter for quite a few months, Verizon is withdrawing the 1-500 service that we use in NH. (Verizon decided not to file a tariff on the product in NH.) As a result, we've been warning users that the 1-500 access number would be going away. That time has come. We expect this number to stop working this week, as early as today. The underlying Nashua numbers (886-6688, 886-7124, 879-9009) will be migrated to our primary access group instead of the trunks that formerly made up the 1-500 service. We expect only minor interruption (if any) while the change is made. In addition, the 227-2100 (Concord) number is also being changed to point to our primary access group. In other words, the Nashua and Concord numbers will continue to work, but via a different mechanism. The 1-500 number will stop working.

November 3 -- web server

The web server was reporting some disk errors over the weekend, all isolated to a single sector. We took down the server briefly this morning for a first pass check (that didn't reveal anything). However after bringing the server back up, we did find the file containing the error and regenerated the file.

Sept 22 -- DSL/V interruption

DSL/V connectivity was interrupted for about 5 minutes starting at around 1pm today. A T3 ATM card experienced a exception and restarted. While it was restarting it was not passing traffic. (Note: this is the first time we've seen this happen.)

Sept 10 -- Paetec T3 down

We have seen increasing loss on our T3 to Paetec over the past few days. After investigation it appears that a T3 card on one of our routers is misbehaving. We have shut down that T3 until we can more fully address the problem. Because even investigating the issue is fairly disruptive we will be doing it in stages over the next few days. Until then all of our Internet traffic will be using our other two T3 circuits.
Status Sept 11 07:00 :The problem on this T3 has been found and corrected, and the T3 is back in service.

Aug 27 -- router configuration affecting DSL

This morning we made a router configuration change to improve the way we handle DSL connections. One positive effect of this change is that DSL connections must be validated by a DHCP discover/offer/request/ack assignment sequence. (This is positive because it prevents people from simply using IP addresses that have not been legitimately assigned to them.) We made every effort to pre-populate the IP assignment information in the router; however some DSL customers may still need to do a full DHCP reassignment sequence (a release and renew).

We apologize for the lack of notice on this activity. Ordinarily we try to give plenty of notice for scheduled changes. However this one had to be made ASAP in order to correct some connectivity issues between DSL users.

We also generally make configuration changes in the small hours of the morning. This change was made in the morning so that any affected DSL users could more readily contact us.

Aug 18/19 -- DSL/V slowdowns

Many DSL/V customers are reporting slow speeds, high ping times, and other symptoms of congestion. We have learned that this issue is affecting large numbers of people who use Verizon's DSL infrastructure (i.e., not just us). Verizon has attributed the congestion to the recent variant of the "sobig" virus (and later to the "Nachi" virus). The slowdown is related to the Verizon DSL network as a whole and not to MV. (In fact our statistics do not show a significant increase in traffic.) Nevertheless please be aware of these virii/worms and take steps to protect and/or disinfect your computers. Some information can be found by following links at the Internet Storm Center site.
Status: Situation has returned to normal as of late Aug 19.

Overnight August 6/7

There will be some configuration work on a core router during the wee hours August 7, 2003. (between 1AM and 4AM). The purpose is to update some configuration elements supporting DSL/V customers. The router may be reloaded one or more times during this period.
Status: The work went beyond 4AM and some of it has been deferred to a later time. One new configuration item would require DSL users to release/renew their DHCP leases, which is too much of a burden without warning. So we'll look for a workaround.

MNH POP July 17 1pm

Power at our Manchester, NH modem POP went out at about 1pm (there was a minor power brownout in the city, and evidently a transformer blew at this particular location.) This means that our small analog modem pool (645-6986) is down as well as some T1 connectivity to other locations in that building. This will be back up as soon as PSNH repairs the power.
Status: Back up at 10pm.

Wed Jun 18, 01:00 (previously announced for Mon Jun 16, 01:00)

The primary http server will be down for a hardware upgrade. Downtime should be around an hour starting at 1am.
Status: Rescheduled for Wed Jun 18
Status: Completed

May 12, 603-886-7124 The Nashua 603-886-7124 number is reporting "disconnected"- this is an old legacy number that we've kept pointing to the newer Nashua service. That newer number (879-9009) is indeed up. We've reported the issue with the 886-7124 number to Verizon and expect them to figure it out and get it back up. In the meantime use the 879-9009 number or refer to our complete access number list.
Status May 13 18:30: Verizon has repaired this number.

No current outages--

but see here for news about dialin number changes, and see here for new (March 7) information about some other backup numbers.

April 17 14:30 - router reboot

After adding a new T3 backbone connection today we did a router reboot. The outage lasted less than 5 minutes.

March 7 17:26 : Usenet News server

The news server is down briefly to address a failing cooling fan. It should be back up by 18:00
Status: it was.

Feb 26/27: BNH dialin numbers are out-- there are new numbers to use.

See here for more information.

Feb 28: Mail slowdown

One of our customer sites is under a mailbomb attack, and a side effect is that our mail server has been slowed down. We are attempting to free up its load. Note: because we recently upgraded the mail server software, we initially were looking for a problem with the upgrade as a reason for the slowdown. This led us in the wrong direction for a while.
Status 20:50 : We've compensated for the mail overload- mail delivery is back to normal.

Feb 25, 2003 1AM -- mail server

The mail server will be down starting around 1AM for the planned mail software conversion. We expect this to last for at least an hour, and it could be several hours. The mail server will be unavailable during this time.
Note: The POP and IMAP capability may be up during this time- do not be alarmed if your mailbox is empty. Part of the installation involves converting your mailbox to a new format. You will not see your old messages until this is complete.
Status 7:15 : The installation is complete although there may be some minor tweaks. The conversion of mailboxes took longer than expected, but the mail server was available during most of that time. Note that we have a lot of complex mail address mappings that were implemented one way under sendmail, and had to be implemented a different way under qmail. While we attempted to cover every special case, it is possible that something slipped through the cracks. Do let us know if you see anything amiss.