Current Outages

March 24- Mail

The mail service at MV has been under an increased load since March 24 (although it started at smaller levels over the weekend). Due to this load you may have experienced some delays or difficulty in sending or receiving mail. In two separate incidents it appears that someone used a forged email address, for a domain that we handle, in some spam runs, which causes many misconfigured mail servers on the internet to send illegitimate blowback mail to our servers. We've taken steps to keep the load manageable, but it is still higher than usual.

Upcoming Scheduled Outages/Maintenance

Nothing scheduled

Recent Events

January 23, 2008, 8:30pm: router

While dealing with the previous router attack issue (see previous entry), we reconnected the router to a different ethernet switch port. The new connection proved to be somewhat faulty, and after lying in wait for a few hours caused a shutdown of the interface until it could be corrected. While access to the Internet was unaffected, access to some of our servers (including nameservers) was blocked during this time.

January 22/23, 2008: router

During parts of Tuesday and Wednesday we were under a sort of denial of service attack from some outside networks. This resulted in sporadic performance problems on a core router. We did not initially identify this as an attack, thinking that there was some other underlying cause, and spent some time trying to find what that might be. However we did find and block the attack sources, which cured the performance problems.

Nov 30, 2007, 12:30pm: Internet

We are experiencing some problems with one of our backbone connections. This is causing problems reaching some sites beyond MV. We are working to fix the problems.
15:48 Note that the "problems" were an attack on one of our customer's servers, swamping our incoming connections. We were able to identify the sources of the attack and had them filtered out by our upstreams. This has relieved the trouble.

Nov 6, 2007: router

Our core router went down around 8:45am and did not reboot on its own. Manual booting was successful and routing resumed as normal around 9:40am. We are continuing to investigate and monitor the situation.

Aug 16, 2007: router

We had several router crashes from about 10PM last night through early afternoon today. This was a result of extra high demand on a customer web site (www.peru.com) triggered by the earthquakes in Peru, and many visitors coming to this site to get information. We did some reconfiguration and applied a bit of rate limiting to protect the router(s) and this appears to have been successful in keeping the routers up and in continuing to allow access to the web site.

July 9, 2007: Internet

For about a half hour in the neighborhood of 10pm tonight, MV users may have experienced data slowdowns or other troubles. What we observed during that time was intense data activity across all of the PVCs (virtual circuits) on our ATM connection to Verizon's network. (This ATM connection is used primarily to connect DSL customers to our network.) We do not yet have any explanation for this data storm.

Wed Jun 6 01:00 2007: primary web services

A short downtime occured while we performed a hardware replacement on the main web server. This planned outage lasted about half an hour as expected.

May 2, 2007: POP and IMAP service

The POP and IMAP service was unresponsive for several hours overnight. This happened as a result of a single SMTP sender that was tying up resources on the server. That sender was blocked.

April 20, 2007: some DSL lines

A group of DSL customers had sync but no routing to the Internet starting early morning. The problem was reported to Verizon who said it was due to the flooding at the Raymond CO. This was so, even though none of the DSL lines were in Raymond; MV maintains a test line at our office in Manchester and it was one of the affected ones. Later, we we told that the loss of routing occurred when Verizon cut over some connection from one routing scheme to another, as a part of their solution to the Raymond problem. It was not necessarily that the equipment itself was part of the flooding. Verizon techs found a problem in how they had done the routing switchover and fixed it around 11pm.
If you experienced this problem and still are not getting connected on your DSL line we suggest you restart your modem and/or router. If you still can't connect after that, call our office and leave a number for callback as we continue to pursue this.

March 16, 2007: web server

The web server copper.mv.net was acting up during the morning and was discovered to have a non-functioning CPU fan. (CPUs are unhappy and prone to error when they are hot.) CPU fan was replaced and the system was back by mid-afternoon.

November 5, 2006: shell server

Shell server crashed (kernel panic) at about 10:30pm, and was brought back up just before midnight.

Mar 14, 2006: news server

The news server was unavailable from about 5AM through 2PM today; it has also been out for a few shorter periods over the last several days. The trouble appears to be an overview database consistency problem. (The overview database is used to give a summary of news articles, so that news reader programs can get and present those summaries quickly.) We ran a recovery operation today, but if this continues may need to do a rebuild of the overview data.

Mar 7, 2006: router reboot

Had a core router spontaneously reboot at about 10:45pm, taking us offline for about 10-15 minutes.

Jan 24 2006: Off-site numbers

Some people reportd problems with the off-site numbers. After connecting, they were not able to reach anything except webpages (ie, email isn't working, etc.). And some customers could not even reach webpages.

We reported this to our provider. In the meantime we suggested you try one of the direct access numbers: MV Communications Dialup Access Numbers

update: seems to have been fixed around noontime.

Nov 18 2005: web server

during the morning hours the main web server will be down in order to increase disk capacity. downtime is expected to last about 1 hour.
update: completed sucessfully

October 26, 2005: shell server

The shell server (shell.mv.net) locked up just before 11pm (looked some kind of kernel deadlock, as simple driver activity was still happening) and had to be rebooted. It was back up at around 11:35.

October 16 : FCG web server (www.fcgnetworks.net)

The FCG Networks webserver is currently unreachable. FCG technical staff is aware of the problem and expects to have it up and running by 6pm today.
Update: The server will be down longer, possibly until morning. FCG webmail users can access their mail via MV's webmail interface which has most of the same functionality. You can reach it at webmail.mv.net and login with your FCG mailbox name and password. For the Server selection, choose the third option: mail.fcgnetworks.net.
Status: The server came back up around midnight. However, FCG users are encouraged to continue to use MV's webmail interface.

October 16 : FCG servers (all but web)

The FCG Networks servers (including the mail server) were offline all morning. The problem was finally isolated to the www.fcgnetwortks.net machine (the main web server) and the other machines are now up and running.

October 8 : server

At around 8:53 one of our servers crashed with a disk error. The affected filesystem had to be recreated and rebuilt; the server was back up at about 13:45. This outage prevented renewal of expired DHCP leases (for DSL), and might have affected the ability of some dialup users to authenticate. Nameservice may also have been slower than usual as a result of clients needing to automatically fall back to another nameserver. UNIX shell customers may also have experienced some problems due to inability to access some network-mounted filesystems.

Sep 26 around 10:00 : dialup numbers and our support lines

The New Hampshire On-Net dialup numbers are not working from some areas, plus some places are not be able to call our support numbers. Other callers have no trouble calling in. It turns out Verizon has a problem with some of their equipment. The story was reported on several local news source webpages.
Status: Lines appear to be working fully as of 3pm.

May 13 15:30 : news server

The news server is down for a short while, to do some things including adding more memory and replacing a cable.
Status: back up as of 17:40

May 11, 2005: web disk

When the power came back on yesterday one of the web server disks would not work. We replaced the disk and began a restore from some oldish backup data. At this point we were able to get the failing disk to spin up again, so we are using that disk as the source of the restore. This should be complete later tonight. Note that only some websites and web spaces were affected.
Status 22:00 All data has been restored from the old failed disk.

May 10, 2005: power outage

Another power outage blackened downtown Manchester today, reportedly due to a substation fire. The power went out at around noon, and came back again a little after 8PM.

Router April 19, 2005

There was a router crash at about 11:40 today, cause unknown, The router simply rebooted itself and was back up in about 10 minutes. Internet connectivity was lost during that period.

News Server April 10, 2005

The news server is down today. It was running in degraded mode (see previous outages entry) while we were building the new server; and it didn't stay up quite as long as we wanted. The new news server has been running and filling up with news, and we had planned to swap it in after the weekend was over. Rather than fuss with bringing the current server back up again, we'll just proceed to bringing the new server online Monday.
Status April 11 1600: The new news server is up and running. The login/password database from the old server has not yet been ported over, so users who had special logins will find that those don't work for a short while. We expect to have these in place in a day or so.

News server: April 3, 2005

The news server is down with a disk failure. No ETR yet.
Status 16:50: back up in degraded mode while rebuilding a disk subsystem. It may be unstable until that is complete, and potentially crash again overnight.
18:40: rebuild failed; the server will be down at least until Monday morning.
April 4 9:30: back up, but again in degraded mode. We'll likely leave it this way while working on constructing the new server.

March 15, 2005: power outage

A fairly large power outage hit downtown Manchester at about 2:30 this morning, putting much of it (and us) in the dark. Power was eventually restored at around 10AM, and we were fully back online as of about 10:40. An outage of this length is longer than we expect our battery backups to last, and extremely rare in this area.

Feb 8 2005: dialup

Calls on one of our digital dialup lines were being refused starting at around 3:30pm. Connecting that line to a spare port on a different server corrected this problem. This does not appear to be the same problem as the morning outage.

Feb 8 2005: dialup

Problems logging into dialup servers were observed in the wee hours of the morning, through about 8:45 when we reset some RAS (remote access server) equipment to correct this issue.

Feb 8 2005: shell server-- not!

Regarding the previous shell server item: after replacing the root hard drive about three weeks ago, the shell server has not had a recurrance of the periodic crashes that we had been seeing. Fingers are still crossed, but that looks like it was the issue.

Jan 17 2005: shell server

The shell server crashed again this morning at about 5AM; it was brought back up around 9AM. We think we are zeroing in on at least one cause of these crashes, but there will likely be more until this cause is proven and corrected.

Jan 8 2005: router

Had some router issues starting at about 1:30AM; a reboot at about 2:00AM corrected this.

Jan 7 2005, 3AM: shell server

The shell server crashed again at around 3AM (two nights in a row) and was down until about 8:15. The cause of the occasional shell server crashes is as yet unknown. We still have more things to try including some additional monitoring.

Jan 6 2005, 3AM: shell server

The shell server crashed at around 3AM, cause unknown. It was back up within about 25 minutes.

Notices from 2004

Listing of events and outages from 2004.

Notices from 2003

Listing of events and outages from 2003.

Notices from 2002

Listing of events and outages from 2002.

Notices from 2001

Listing of events and outages from 2001.

Notices from 2000

Listing of events and outages from 2000.

Notices from 1999

Listing of events and outages from 1999.

Notices from 1998

Listing of events and outages from 1998.

Notices from 1997

Listing of events and outages from 1997.

Notices from 1996

Listing of events and outages from 1996.