Never a dull moment

Not a week goes by without some kind of emergency: hackers, backup server woes, operating system issues, hardware trouble, software trouble, spammers, integration of new technologies. Round and round it goes.

So, to start with Hackers. A long long time ago, EBOUNDHOST acquired a smaller hosting outfit to broaden its offerings with cPanel. Up to 2005, EBOUNDHOST was a Plesk only outfit. cPanel and Plesk are two competing hosting control panel systems that run on Unix-like servers. Both systems have their raison d’être, one is better suited for power users, another for SOHO and non professional website owners.

Unfortunately, one of the acquired cPanel servers had a serious vulnerability which was inherited with the machine. The root-kit survived our admins’ sweeps and lock downs and lay dormant for at least a year. When the time was right, our friendly hacker, or should I say `cracker` (hackers generally don’t damage systems), sprung into action. When the situation was finally under control, several clients no longer had their databases and files were missing. Unfortunately, the attack was scheduled in a time when the server backup was in progress and corrupted the backup. This was a glaring oversight, and our team took ownership of the problem and helped our customers rebuild their websites from old backups with the help of pieces recovered by the Data Recovery procedure.

After this event, a new backup strategy was deployed to production almost immediately. Client data is now archived in snapshot style for several weeks on our new Backup Server cluster. All of our Shared servers and many Dedicated/VPS hosting customers make daily backups to this system. Additionally, databases are being archived in a separate structure. Whereas previously, to recover a single client, an entire server backup had to be unwrapped onto a dedicated machine and then moved back into place; one of our techs can now mount the image and copy the files back into place within minutes. This is possible due to some very cool technologies that became available recently, but this is geek talk.

In real world terms, this requires a tremendous amount of storage, lots of spinning hard drives RAIDed together into mammoth terrabytes of backup space. But it’s never a dull moment, just two months later, we’re almost out of room.

Not all emergencies are of the bad kind some are just exciting opportunities, but one common thread emerges, in hindsight they are all valuable learning experiences. Once you pass one hurdle, the next one seems more approachable.

Artur June 14, 2007