About servers. Everyone reading this post is making a connection to a server. In fact, you are making a connection to at least a couple. There is a server in your office or home that is allowing you to proxy onto the internet, most likely a wireless router, which connects through another server, the DSL or cable modem. There is a caching DNS resolver server on your ISP. An entire army of router servers between your home and our data center. And the last server in the chain is our web server, which actually hosts this content.

Lets narrow down the definition of a Server. We are not going to talk about IBM mainframes or Sun UltraSPARC based blade systems. Today, we care only about the servers which comprise the majority of the infrastructure of the websites you visit. These are normal computers just like you have in your house or office, with the exception of being confined into more efficient packaging. They use familiar Intel or AMD processors, normal DDR2 or faster RAM, and SATA hard drives. What really separates them from home PC’s is the software. But software is not what this blog is about.

Here is what one of our older servers looks like (below):

To compare, here is one of eBoundHost’s newest servers.  This form factor is unofficially called the ‘pizza box’ due to its small dimensions.

The first thing you will notice is that the new server is not as tall. Our older hardware uses 2u (units of space) while the new servers use 1u. This allows for greater density. Some servers use as much as 7u but these are specialty machines that are filled to the brim with hard drives in gigantic RAID arrays.

Side to side comparison:

These servers fit into specialty (read expensive) racks that have 42u of storage in each rack. This means that when filled with 2u servers, we can only install 21 machines instead of 42 1u servers. It’s a dramatic difference when you talk about a server room full of racks such as in our facility:

Of course the entire 42 units are not available for servers, there are switches, power distribution units, firewalls, intrusion detection equipment.  All considered, we are happy to have 30 servers in one rack.

There is also the consideration of electricity and heat. A rack full of servers eats electricity like a hungry SUV, and produces just as much heat pollution. 30 servers stacked on top of each other, blowing air into the same direction, require an amazing amount of cooling, which needs big air conditioners that move a lot of tonnes of air. That’s all I’m going to say about that. Data center challenges is going to be saved for another blog entry.

To jump back into server hardware. Here is the same 1u server without its cover.

Motherboard, CPU, heatsink, RAM, hard drive and a very powerful cooling fan. Seems simple enough. Another picture:

Every server is custom built. When an older machine comes off line, we generally sell it through eBay and build a new server to take its place. The nature of hardware is such that components wear out and fail eventually. Our clients and our reputation are far too important, so we give old hardware the boot and use all new equipment.

Here are some servers in action. The following pictures may not be completely safe for geeks, they may cause weakening of the knees and a desire to run out and fix something. Please refrain, it will pass:

These (above) are dedicated servers. Inventory tags have been obfuscated in order to protect the innocent.

(below) Are some specialty machines which have 15k SAS (fast/expensive) hot swappable hard drives in RAID array. Used for our shared servers, VPS machines, and some powerful dedicated servers.

Each server is built by our staff. We love them so much that we have hundreds of them

More to follow, there is so much to cover: data center, operating systems, server software.

Affiliates have been receiving $100 for each hosting account referral since the beginning of 2007. However, during a staff meeting last month (Nov), it was decided to raise the payout and make it more profitable to refer business to eBoundHost.

Shared hosting accounts (Home and Professional) remain at $100 per referral. However, VPS and Dedicated servers, are now their own respective categories and have been raised to $200 and $300 per referral.  The system works the same way, someone clicks through the affiliate link, orders a VPS and you receive the commission.

There are more details on the affiliate page, but in a nutshell this is the big news:

Hosting: $100
VPS: $200
Dedicated: $300

Keep up the good work and we hope all of you have a wonderful holiday season.

BTW, past referrals are not eligible for this payout, so don’t even ask!

The Thanksgiving holiday is almost upon us. From the eBoundHost crew, I would like to wish all our friends who are observing this wonderful day, to have a good celebration and try not to have too much turkey (or whatever else).

This is the one holiday per year that seems completely innocent and not commercialized. Its spirit has somehow been preserved over the years and has not become a day to give cards or mandatory gifts, just to get together with your family around a dinner table and enjoy each other’s company.

On a related note, some of our staff are traveling around the country in the next few days, so we are running on a skeleton crew. If support runs a bit slower than usual, we hope you understand! (non critical/outage tickets only, of course)

So without any further delay, happy Thanksgiving.

Recently i have been following some very interesting conversations in the Free Open Source Software (FOSS or just OS) community. In case you have been on the moon for the last 10 years and have not had any news updates, I’ll fill you in. There are entire communities of people who are continually building all types of software that they distribute for free on the Internet. These people are computer programmers, graphic artists, copywriters and others. The software they build ranges from an operating system like GNU/Linux or to the online encyclopedia Wikipedia, and my personal favorite, TiVo.These projects are built by mostly unpaid programmers who contribute their free time and knowledge to build a better program/operating system/encyclopedia. They are driven by an altruistic desire to build a better “whatever.” They, then release this “whatever” to the masses for free, who in turn take these programs and hopefully make money from their distribution or by providing support. Ideally, whenever someone makes money from these projects, they will turn around and support the project they are using, thereby supporting the programmers who in turn will be able to make an even better program. So this is an ongoing cycle with contributions helping to fund development.

Sometimes these free programs are backed by large companies such as Sun (creators of Java) or TiVo. They often distribute a core system free of charge and hope to get a user base hooked on their product so they could then sell them the advanced software with more robust features. And sometimes companies find it easier to find a open source project and build their own system on top of it. For example, there are wonderfully powerful SQL systems available for free, MySQL and PostgreSQL. They have been in development for 7+ years and are such powerful systems that the vast majority of today’s Internet applications are based on one or the other. It would be foolish for a small (or large) company to start building a product from the very ground up.

The potential of such integration is HUGE. Company A only needs to make sure that their product integrates with the FOSS project B. Company A does not need to worry about any potential security threats or unexpected crashes due to project B. Instead, the programmers for project B take care of all such issues. This way company A is able to focus on improving the usability of their own product, which, coincidentally, does not even have be FOSS.

So this gets us to the really interesting part, sometimes Open Source projects are not compatible with each other. You would think that these programmers would be smart enough to allow these programs to integrate, but the problem is not what you think. Sometimes their licenses are incompatible!

The two licenses that I’m most concerned about are the GPL and BSD licenses. The BSD license says “do whatever you want with this code” which means that you are free to take code and even distribute it as closed source, proprietary programs. Coincidentally, the Apple X operating system is based on FreeBSD with some (very major) changes. So according to this license, Apple is able to take the current FreeBSD code, change whatever they want to change and distribute it as an independent project. Thankfully, Apple was kind enough to contribute significant improvements back to the project, but they did not have to. The BSD license allows them complete independence. The company does not have to release its trade secrets, only what they chose to. However, it is in their own best interest to contribute back and make sure that FreeBSD continues to be a vibrant operating system. This way, for the next release of the Apple software, they simply grab the FreeBSD code base, apply their own changes to it, and they have an up to date system with full security patches.

So the point of this blog is the other, and probably more important license, the GPL. This one says, that you are free to distribute this program as long as you make all the changes available to the public. So any company that modifies the source code of any project, thereby improving it, will have to open up its changes (including trade secrets) to the greater community, and if these changes are quality, they are then going to be integrated into the main project. Linux is distributed under this license. The added complexity of GPL is that there are 2 separate GPL’s on the table today GPLv2 and GPLv3. The v3 is newer, more complicated and puts restrictions on how the code can be used. We are slowly making the transition from v2 to v3 but a lot of companies are unable to make this transition because they would have to open up their closed source addons to GPL’d software, which were allowed under v2. So companies have to make a choice, use old and buggy GPLv2 software or upgrade to GPLv3 and lose business. This is a big issue that is beyond the scope of this blog, and we’ll see how things work out in the coming years.

So as far as we’re concerned, the major difference between the two licenses are

BSD: you should do the Right Thing but we won’t force you
GPL: you have to do the Right Thing and contribute back to the project

As a active reader of the FreeBSD mailing list and Slasdot and various Linux lists, I have recently started to see a lot of chatter about how the GPLv3 license is superior to everything else. People froth at mouth (or keyboard) and spew such hatred towards the other camp. After reading through 10 pages of this drivel, I realized that these people are wrong. They are trying to make a better FREE license by putting more restrictions on how it can be used. This is ridiculous.

Seems to me that they forgot what all of this is about and are trying to bite the hand that feeds them. If you take out corporate money out of open source software, a lot of important projects will collapse. The real support ($$$) comes from corporations that support projects which help to make money. No corporate profit means no support. Live and let live, if someone makes good money using FOSS, more power to them!

Last time I checked, making money is good for the person/company making money and for the entire community which benefits from the added support, either through sponsorship or awareness which leads to more pubic support. Companies like Google, have entire teams of programmers working on Open Source software and contribute millions to improving other ongoing projects.

Not to mention that you simply cannot outlaw capitalism which is what they seem to be trying to do. If people are forced into the corner, they will find another exit. And guess what, the BSD license is not a bad alternative to GPL’d software. If the GPL people keep pushing, they will just drive away developers to the other camp.

FOSS is great for all of us. Everyone should step back, take a deep breath and refocus on building better software rather than bickering about nonsense.

Oh, and for the record, this entire document is entirely not open for redistribution without my permission. How is that for a license

Thursday august 23rd, 2007, at approximately 11 pm, our backup power generator was unable to cope with a lightning strike to our data center and shut itself down.  A technician was dispatched from the generator manufacturer, CAT, and was on site in 15 minutes after the outage.  Battery power allowed our servers to keep working, however, the air conditioning units were disabled until generator power could be restored.  After approximately 20 minutes of battery power a decision was made to bring down the equipment in order to avoid heat damage.  Approximately at the same time as the network was being brought down, two mobile power generators capable of producing 1 Megawatt each, arrived on site on flatbed trucks.

After the two mobile generators started to feed the air conditioners and our main building generator kicked in, all servers were turned up to full service and were scanned for array problems. All together, most customers experienced about 30 minutes of outage.

In the last 24 hours, the Chicago area experienced floods, severe thunderstorms, and a tornado.  Trees are broken everywhere and one of our staff lost his car to a flood.  All things considered, we are very lucky that the damage was not any worse.

To head off any questions, yes we do have backup batteries, yes we do test the generator every month, and yes, we are prepared and have survived times this type of situation many times.  We apologize for the outage and will be happy to speak with you on an individual basis about how this effected your service and what we can do to help.

Not a week goes by without some kind of emergency: hackers, backup server woes, operating system issues, hardware trouble, software trouble, spammers, integration of new technologies. Round and round it goes.

So, to start with Hackers. A long long time ago, EBOUNDHOST acquired a smaller hosting outfit to broaden its offerings with cPanel. Up to 2005, EBOUNDHOST was a Plesk only outfit. cPanel and Plesk are two competing hosting control panel systems that run on Unix-like servers. Both systems have their raison d’être, one is better suited for power users, another for SOHO and non professional website owners.

Unfortunately, one of the acquired cPanel servers had a serious vulnerability which was inherited with the machine. The root-kit survived our admins’ sweeps and lock downs and lay dormant for at least a year. When the time was right, our friendly hacker, or should I say `cracker` (hackers generally don’t damage systems), sprung into action. When the situation was finally under control, several clients no longer had their databases and files were missing. Unfortunately, the attack was scheduled in a time when the server backup was in progress and corrupted the backup. This was a glaring oversight, and our team took ownership of the problem and helped our customers rebuild their websites from old backups with the help of pieces recovered by the Data Recovery procedure.

After this event, a new backup strategy was deployed to production almost immediately. Client data is now archived in snapshot style for several weeks on our new Backup Server cluster. All of our Shared servers and many Dedicated/VPS hosting customers make daily backups to this system. Additionally, databases are being archived in a separate structure. Whereas previously, to recover a single client, an entire server backup had to be unwrapped onto a dedicated machine and then moved back into place; one of our techs can now mount the image and copy the files back into place within minutes. This is possible due to some very cool technologies that became available recently, but this is geek talk.

In real world terms, this requires a tremendous amount of storage, lots of spinning hard drives RAIDed together into mammoth terrabytes of backup space. But it’s never a dull moment, just two months later, we’re almost out of room.

Not all emergencies are of the bad kind some are just exciting opportunities, but one common thread emerges, in hindsight they are all valuable learning experiences. Once you pass one hurdle, the next one seems more approachable.

server ORANGE is having serious issues with it’s RAID array, hardware is being replaced and rebuilt. Techs are working on it as I’m writing this message. There may be a series of brief outages, but the system should be completely solid during business hours tomorrow morning.

It is 10:35 PM CST on Thursday, February 2nd, 2007

Server ORANGE is going to be brought down for emergency hardware maintenance in two hours (Feb 15, 2007 @ 1pm). The outage should last for no more than 20 minutes, possibly less.

My apologies for doing this in the middle of the business hours but my tech says it’s a real emergency.

Last night, January 25th, 2007, there was a brief outage that effected our entire network at 2 am - 2:30 am.

Our network people had to take down both of the redundant gateway switches for maintenance (big iron 1 and big iron 2) resulting in all external traffic completely stopping. The switches were dropping packets and had to have some hardware replaced.
Some of our customers did not know what this was a scheduled outage and we were flooded with email as soon as the services were back up. So I want to take this time to point out the Network Status page which has a running schedule of all upcoming maintenance (network and server).

The vast majority of such repairs do not result in outages, since just one switch at a time is usually being worked on, but this was a very special event. They explained why both of the switches had to be worked on at the same time, but not being a network person, it was a bit over my head.

Happy to report that the eBoundHost affiliate system is picking up momentum nicely. We have been running for just over 160 days and the statistics speak for themselves. A lot of people are making quite a bit of money very quickly. The great thing is that there is almost no work involved, most of these people simply place links on their existing websites and the system takes care of the rest. Click here to view affiliate progress.

So what is the best way to make a great affiliate system even better? Raise the payout! How about $100 commission per customer? It seems outrageous to pay more in commission than the client brings in over a year but it is completely sustainable for the eBoundHost business model. We are here for the long haul.

To all of our affiliates: we are thrilled to work with you. Keep up the good work!