One rough Monday

So while last Friday went without any problems, today is Monday so something was to be expected. No wonder that it is supposedly the most common day of the week for heart attacks.

As the day began and all system components began stressing out with highest load of the week (monday morning) the weakest link failed, the power unit. All 4 RAID drives were spinning and CPU usage (and power consumption) shot through the roof as the email began to pour in and tens of thousands of little files were being written to the mail system. But to be fair, these things happen, nobody is immune from hardware failure, this is why I always stress having a full backup of all your content.

In little less than an hour, everything was back to normal. RAID array checked out, file system checked out, temperature nominal on the new power unit. All was well until the hour-long outage began to catch up with us in the form of an avalance of email that was not received on time. When it began to pour in, it really hit hard.

The bad news is that it took a few hours to work through the queue and there was a delay with real-time messages.  The good news is that as a result of this experience a new mail system is being implemented that will eliminate 90% of all spam before it arrives at the hosting servers. This mail will be filtered out as it hits the network and will reduce the stress on machines that have other tasks to handle such as serving websites.

Testing will begin early next week and we should be fully implemented before end of the month.

Artur October 16, 2006