Thursday, 11:00 am November 4, 2004

Our mail servers crashed yesterday afternoon due to what appears at this time to be a local spammer attack. The mail servers are back up and running at this time, but the webmail applications are not working. We will be working through the back log of mail today to filter out the spam and get it injected back into the system.

Tuesday 11:15 am September 28, 2004

One of our core routers crashed this morning when we inserted a DS3 card into the router. This caused an interruption to the local network. We have rebooted the router and everything is functioning as expected.

Tuesday 9 a.m. September 21, 2004

Our primary mail server stopped processing messages through the anti-spam system last night due to a memory leak. We have restarted the server and are working with the vendor to address the leak. The backlogged mail in the incoming queue is processing through the anti-spam system at this time.

Monday 1 p.m. September 20, 2004

We have installed equipment at our new location in the Level(3) Network facility in San Antonio. We are waiting on completion of a Fast Ethernet link into the Level(3) Network to add this connection to our BGP routing tables. The projected completion is by Wednesday, September 22.

Monday 3:00 p.m. August 2, 2004

Our connection to Savvis is experiencing routing problems again. We are in communication with their engineers concerning why the BGP session is being interrupted at this time.

Monday 1:30 p.m. August 2, 2004

Our incoming links to both our internet connections went down due to routing problems earlier this afternoon. They are both up again and we are working with the providers to determine the source of the problem.

Wednesday 6:30 p.m. July 21, 2004

We lost several of our bandwidth links this morning causing us to move to a new provider on very short notice. We have installed new links and are in the process of getting the bandwidth load balanced. We will be adding additional bandwidth as we resolve some problems with one link to the new provider.

We reduced the allowed bandwidth per user today to preserve the available resources. The reductions will be removed later this evening as the new circuits prove themselves reliable.

Monday 3:15 p.m. June 21, 2004

We have installed the new mail server and it is processing mail as expected. We are copying the existing mail store over from the old server at this time, so new mail will come in immediately but older mail will be in the mailboxes in a few hours. We currently have webmail disabled because we can't merge the webmailboxes without losing data. It should be online within an hour.

Friday 1:00 a.m. June 18, 2004

After extensive analysis we have found the mail server is experiencing a performance degradation in the drive sub-system that appears to be getting worse. The hardware all reports no errors but the effect is real. We are expecting to receive a replacement server today and put it into service over the weekend. During that time the primary focus will be on the reliable delivery of mail. The spam quarantine and related functions are a secondary concern as we are trying to reduce disk access until the new server is in place.

Wednesday 1:00 p.m. June 16, 2004

The mail server is currently overloaded with incoming spam which is causing a delivery backlog. are working on the problem at this time.

Tuesday 12:00 p.m. June 8, 2004

The web server is back up and serving pages. There is considerable configuration remaining to be done in order to allow editing of web sites via Frontpage or FTP.

Tuesday 4:00 a.m. June 8, 2004

The web server is down for maintenance.

Monday 10:30 a.m. May 3, 2004

The inbound mail is currently backed up due to a spam attack from a client's server that has been compromised. The client's server has been quarantined from the service and we are working through the mail queues to delete the spam and recover legitimate mail.

Sunday, May 2, 2004

We experienced problems in our core network Sunday morning related to the Sasser virus outbreak. The volume of the virus attack on the core subnet was overwhelming the servers and their ability to communicate with the routers. We resolved have resolved the problem.

Tuesday, 1:00 a.m. March 10, 2004

The mail server has been upgraded with a new version this morning for better spam detection and resistance to rogue mail attacks.

Saturday, 8:00 p.m. March 6, 2004

We have just recovered our server from what appears to be a spam attack from one of our own customers. We are reviewing the logs at this time and have quarantined a large quantity of mail and are going through it at this time to determine exactly what happened. The mail server is now up and the offending customer has been disconnected.

Wednesday, 3:45 p.m. January 28, 2004

Savvis corrected a problem with our connection and it is now back to full capacity. There was a short outage forcing all our traffic through Sprint, but connections have now returned to normal.

Tuesday, 9:30 p.m. January, 27, 2004

Our primary web server's motherboard failed around 4:45 p.m. this afternoon and could not be recovered. We have moved the data to a new server and completed the rebuild. The rebuild took longer than we would have liked because the new hardware is substantially different than the original server (different processors, different motherboard, more memory, bigger disks, etc.). We will continue to monitor the new server to assure it is stable.

Tuesday, 11:00 a.m. January 27, 2004

Savvis is reporting problems on our connection to them that is affecting our connection speed. They are working on the problem at this time. We do still have a connection but it is somewhat degraded in capability.

Saturday, 12:30 a.m. January 24, 2004

Our primary web server had a serious drive failure this morning sometime before 10:00 a.m. We have now recovered from the failure but there are issues with a number of websites we host. Due to the rapid recovery process a direct restore from tape was not possible without overwriting other critical information concerning the system. A slower process would resolve this but would require waiting on hardware that is not locally available. We will begin the process of correcting the current configuration to restore all the websites ASAP.

Wednesday, 10:30 a.m. January 22, 2004

While upgrading our mail server to a new version to provide additional anti-spam functionality we have experienced a failure in the smtp-auth protocol. The vendor is working on the issue at this time. Smtp-auth is the functionality that allows a customer on an external network use our mail system as their outbound server by authenticating the connection using the customers email address and password.

Update 11:00 a.m.

We are regressing to the previous version to await a resolution of the smtp-auth problem.

Update 1:30 p.m.

We have upgraded again and installed the latest patch from our mail software vendor. The smtp-auth problem has been resolved.