2002 outages/events

Tue Dec 3 @ 1:00

News server will be offline for aprox 15 minutes.

Monday November 18 @ 09:00am

Email failing for some nodenames: email for clients with long sitenames is failing. Tech support is working on this now.
Status approx 11:00AM This has been fixed.

Oct 30ish, 886-6688

The old Nashua hunt group number 886-6688 stopped working. This is a very old number that used to be used in the original Nashua modem pool. Eventually the Nashua modem pool was completely replaced by digital trunks and a new number was assigned, but we kept the old 886-6688 number in addition to the new number. Some years later the digital dialin pool was converted yet again to another trunk group and yet another number was assigned: we kept the original number and the second number working and pointing to the new trunk group. Evidently the paper trail got tangled and the original number was assigned to a new Verizon customer and it stopped working for us. Verizon has now repaired this problem and the number is working again. We do recommend, though, that you use the newer numbers as listed on our access numbers page.

October 17/18 overnight

Around midnight the morning of October 18 (that's the night of the 17th into the morning of the 18th) one of our telecomm suppliers will be moving some of our circuits into a new distribution facility. This will affect all of the BNH dialin circuits as they are moved one at a time (corresponding to 23 dialin lines at a time), and it will also affect our T3 that connects us to our DSL/C customers. The outage on each circuit will only be a few minutes- the total work period will be around 1-2 hours. We will also take this opportunity to do a minor modification to one of our core routers-- this will involve one or two 10-minute outages.
Status 1:00AM The circuit moves completed successfully tho the DS3 was down a little longer than we thought. However all work was completed within about an hour.

Sat Oct 12 01:00 EDT 2002: primary web server

Planned memory upgrade for web server. Outtage should last no longer than 20 minutes or so.
Status Sat Oct 12 03:27: Success. Better late than never.

Aug 28 2002: servers

A disk drive hung on the web server this morning, forcing a reboot of both the web server system and the mail server. This has happened a few times over the last several weeks, so we'll be looking into options to correct it.

Aug 26 2002: 1-800 support number

The 1-800-MVC-NETS support number went down today due to an apparent carrier-wide outage. The direct number (603-629-0000) continues to function during the outage.

Wed Aug 21 01:00 EDT 2002 -- shell server disk replacement

We will be replacing a disk in our shell server during the early morning Wednesday. The disk has reported some non-fatal errors, but as a precaution we are swapping in a replacement disk. Also the new disk will provide additional space if it becomes needed at some point. Downtime should be no longer than an hour.
Status Wed Aug 21 03:00: Apparent success.

June 20/21 2002 overnight -- webserver adjustments

We worked on optimizing some configurations on our web server, including:

• Reworking configuration files;
• Tweaking and adding some tuning parameters;
• combining the standard HTTP server and the secure HTTPS server, which had previously been running as seperate processes using different configuration parameters.

While this doesn't really represent an outage, web users may have noticed a few short pauses in the availability of the server while it was restarted. Also, if you notice any changes in the behaviour of the server, please let us know.

Thu May 30 01:00 EDT 2002 -- webserver disk upgrade

Our primary webserver will receive a new much larger disk to replace an existing much smaller one. The expected downtime should be no more than half an hour. In addition, the powersupply will be swapped with one of a higher wattage capacity, and a possibly failing cpu fan examined.
Status Thursday, May 30 02:20 EDT: Almost an hour later than we hoped, but everything went successfully.

May 24 2002 -- Manchester POP outage

The power is out at our Manchester POP (Point of Presence). This is a separate small group of phone lines and analog modems that gives Manchester and surrounding towns an alternative dialin pool. If you normally dial this modem pool (at 645-4986 or at another number starting with 645, 622, 624, or 634) please refer to our main modem numbers for an alternative. Make sure you verify that the number you dial is local to you.
We have no information yet on when power will be restored.
Status Tuesday, May 28: Power was restored at about 9:20 this morning.

Monday May 13 @ 2:15pm -- shell server failure

Our shell server (shell.mv.net) is currently down.
Status 4:10pm: It appears the power supply has failed. We are working on restoring the machine now and will be back up as soon as possible.
Status 7pm: Looks like a double whammy. The power supply had indeed failed but the root disk is also corrupt. We will be working on either repairing or replacing this disk. The shell server is likely to be down until midnight at least.
Status May 14 1:15 AM: System is back up. Since we needed to rebuild the root disk we also took the occasion to upgrade the operating system. Shell users please let us know if you experience any problems as a result.

Monday April 29 @ 01:00am -- core router reconfiguration

One of our core routers will be taken down for reconfiguration during the early morning Monday. An expected effect of this will be a virtually total network outtage, which should last for less than an hour.
Status: Complete: maintenance took approximately 10 minutes.

April 16, 2002 -- email to mindspring.com

mindspring.com mail servers are inexplicably rejecting any email forwarded from our main mail server. While we are awaiting some sort of response from mindspring's technical contacts, we have configured our main mail server to send mail destined for mindspring.com over to a secondary mail server here at MV that is not being blocked by mindspring.com . This is a temporary solution that may not be stable but hopefully it will last until we get the underlying issue explained and fixed.
Status: This issue is also affecting earthlink addresses; we have applied the same workaround to those.

Earthlink has yet to respond- your correspondents who have mindspring and earthlink addresses might be interested to know that legitimate email to them via our primary mail server is being blocked for no apparent reason, without warning, and without response. Perhaps if those customers alert their provider to this situation it will be more productive: specifically that email from the mail server mercury.mv.net at 199.125.85.40 is being rejected without cause.

April 17: As it turns out, our mail server was vulnerable to a somewhat archane form of mail relaying. We did hear one response from Earthlink (finally) saying that they had performed a relay test on our server and it failed. They would not or did not tell us what the specific test was. Since we have protected our mail servers from relaying unwanted mail for many years, we would have been interested in being informed about the test. We were able to locate the test in our logs and confirmed the vulnerability, which has now been patched. While we did have the flaw (keeping up with new exploits is a little like keeping up with viruses), we have to say that that Earthlink's policy of test and blacklist without providing the least bit of notification, plus lack of informative response (indeed, mostly lack of response at all) is not helpful.

April 17 20:00 : Earthlink gave us a brief reply that they retested our server took it off of their blacklist. There was still no information imparted to us about what specific test failed although as we said we were able to figure it out from our logs. No other relay tests that we have found test the particular exploit that got through-- indeed it was good to find out about it. We'll keep our workaround in place overnight at least, and remove it tomorrow.

Fri Apr 5 04:00 EST 2002 -- switch upgrades

This is a continuation of the previous days switch upgrades.
Status: success. The unit that was resisting its firmware update was sucessfully upgraded using an alternative method.

Thu Apr 4 04:00 EST 2002 -- switch upgrades

We will be performing some routine firmware upgrades to a couple of our core switches. An brief (approximately 2 minutes) expected loss of connectivity is expected.
Status: some success. One unit encountered some trouble accepting the upgrade. A second attempt is scheduled for tomorrow.

Feb 15 2002 -- Mail server backlog

Today we discovered a significant backlog on our mail server that evidently had been in effect for at least a week. Some mail may have been delayed or even lost as a result. There were several causes: significantly one dedicated customer who elected to turn off their SMTP server apparently because of being mailbombed, and let the mail simply accumulate in our queue instead. The increase in spam and replies to mail with bad return addresses also contributed to an increased queue size. Also as you know we did a recent major upgrade to the mail server hardware but had not tuned the mail delivery program to account for the higher capability-- we have now adjusted some of the parameters to let it deal with larger queue sizes and let it run more processes when delivering mail out of the queue.