Several VPS servers were down tonight for software and hardware maintenance, if you did not receive a notification ahead of the maintenance, your system was not effected, feel free to stop reading.  Everything is now back online and arrays are happily rebuilding.  Most servers experienced 15-20 minutes of total downtime but one server in particular took over 45 minutes to finish the process.

Beyond the outage itself, customers were not impacted.  No data was lost and there is nothing you need to do post upgrade.

Great success!

Its November 4th, election day in the USA, and all the nonstop political coverage is almost over. Whatever your political leaning, I’m sure we all agree on one important thing, electing our leaders is one of the few true freedoms we have. However you vote, its better than failing to show up. If you don’t like either candidate, then write in your own name. This is a much better way to protest than through aparent apathy. Howvever you vote, just vote.

Let’s hope that whomever takes the White House is going to unite the country that has been so bitterly divided politically for the last few election cycles.

As life gifts us with happy moments, we must always remember the tragedies of our past that shape our future.  9/11/2001, always in our hearts.

We were up and down sporadically this morning for about 15 minutes.  The network issue took a little bit of time to clear up, but we should not see any more downtime for the rest of the day.

Someone emailed me these pictures and could not help but post them up for your amusement! This is the very opposite of how a data center is supposed to look.

For a very good reason this article is being written and published entirely on my blackberry.

If nothing else it proves the point that you never know how your website is going to be accessed.

Like it or not, there is now a whole new dimension of compatibility to test for when designing your new website.

Having just come back from the HostingCon 2008 industry trade show, my head is still buzzing from all the presentations.  Everyone from hardware vendors (ICC-USA) to the great satan himself (Microsoft) was presenting their wares and as much as I hate to admit it, even the microsoft presentation was very impressive.

There was a lot to see this year from the vendors, but far more interesting were the presentations and group learning sessions.  Far less time was spent on the technical aspect this year than in previous years.  There were no “how to install XEN paravirtualization” classes;  instead, many sessions focused on the business itself, such as graphically mapping the company’s track record, how to evaluate your hosting company’s net worth, and some very interesting QA sessions with industry leaders who spoke about managing growth and valuation of assets.  Very grown-up conversations indeed!

The thing I noticed more this time than in past years is how the keywords “cloud computing” and “grid hosting” were thrown around the room.  It’s the new Web 3.0 terms that have had little meaning in the past but are now, all of the sudden, more tangible.

So what’s the deal? Grid/Cloud computing means: a “process/computation” moved off a single server into a “cloud” of computers.  A group of servers (can be tens of thousands) is presented as a single unit to handle a task with the combined power of all the processors/ram/storage.  In this model, all systems are the same when it comes to both hardware and software, and are completely interchangeable. Meaning that you can take any two systems and swap their physical locations in the cloud, they will not need to be reconfigured.  All systems perform the same task.

While this model works very well for parallelizable tasks such as graphics processing and mathmatical computation, it simply does not work for hosting a website.  A “grid” or “cloud” as they are being presented are far less useful than a “cluster” of servers.  While I may be too picky, I feel terminology is very important.  In our case, a Cluster is a group of systems, each performing a specialized task (web, mysql, dns, email), and presenting a unified interface to the Internet.  Here, you cannot take a server from the MySQL group and swap it with the server from the eMail group and expect them to work.  They are very distinct systems with differing hardware and software configurations.

This “cluster” model is a very old concept and is the only way to host the largest websites, such as facebook, myspace, youtube, google, yahoo.  When the existing systems are approaching saturation point and load spirals higher and higher, you simply add another front-end machine to the effected segment of the cluster to offload some of the processing.  When web traffic goes up, add another front end apache server.  Too many SQL queries, add another mysql machine.  In essense a Cluster is a collection of Clouds/Grids.  Each Cloud handles its specialized task and contributes to the performance of the main Cluster.

What I find infuriating is that some providers are talking about Cloud or Grid computing as though it’s the next step in VPS hosting.  This is so misleading that it makes my teeth grind.  There is no way to run a single VPS instance over a cluster/cloud/grid of computers.

When they market their VPS service this way, it makes a client believe that if the server that hosts their VPS, has a meltdown, their own system will continue to run on the rest of the cloud without interruption.  In fact, what happens is that a crashed system is a crashed system, and your VPS instance will also go down in smoke with the rest of the server.  And while it may be restarted almost immediately, it will still have downtime.

Also, they claim that you can scale out your VPS to unlimited levels, implying that it’s a trivial task to add more processing power.  The way they handle this is by adding other VPS instances of the same system and splitting the traffic with a load balancer.  This has it’s own tremendous issues because you cannot take a normal website, split it into two or more instances and expect it to function properly.  Websites have to be designed especially to handle this scenario.  For instance, MySQL files cannot be written to at the same time by two instances of MySQL without experiencing some very serious corruption.

More than that, this new “Cloud” model is billable based on usage of cpu cycles, bandwidth, disk access.  This means that you never really know how much it’s going to cost at the end of the month.  This is especially wonderful in the case of a Denial of Service attack which can burn through server resources like there is no tomorrow.

Today’s providers who claim to live in the Cloud, are using traditional hosting technology masked with a very fancy control panel.  In my opinion, cloud computing still has a long way to go before it’s going to be useful for our industry.  Today this technology is useful to a handful of customers.  For the rest of us, we have Shared hosting, VPS hosting, and Dedicated hosting.  With the incredible pace of technological advancement of individual servers, I don’t see a reason to move to the cloud.  Today’s web servers are more powerful than yesterdays supercomputers and this trend will continue for a long time yet.

Last thought, when the Cloud is going to become useful, we will add it to our product line.  For now, our products are every bit as useful as anything available today on the market, and without any fancy buzz words.

It’s good to be popular but it definitely comes with it’s own problems.  For instance, today, some clever folks decided to run a Distributed Denial of Service attack on the eBoundHost.com domain name.  They knocked us out of the web for a little bit of time, but luckily our monitoring system sounded an alarm and a tech was dispatched to fix the problem.

What happened?  A standard server simply cannot cope with several hundred servers trying to access a website at the same moment.  At first things work fine, then they slow down and finally, the server runs out of allowed processes.  The Apache web server is now effectively useless, hence the title: Denial of Service attack.

How does the attack happen?  Someone’s grandmother receives an email on her AOL account that promises to have pictures of her favorite relatives.  She opens the picture only to infect her computer with the most nasty Trojan virus known to mankind.  This Trojan proceeds to let his friends know that there is a party happening at grandmas house.  They come to visit and also infect the computer.  All sorts of fun things can be installed this way, for instance, software that turns this computer into a node on a botnet.  This botnet zombie is now fully in control of some 16 year old in Vietnam/Russia/Turkey/etc, and this computer can now participate in things like sending spam or a Denial of Service attack.

There are definitely ways to deal with this kind of situation.  First off, there are devices you can buy that deal with known DDoS patterns.  There are lists of known zombie ip addresses that you can block out on the router.  There are ways to deal with this type of situation.  Luckily this does not have all that often, and it is usually enough to merely let the attack work itself out.

This time the attackers were nice enough to have left us a signature of their work, and for that we are very grateful.  It really made the cleanup effort much easier.  So I wanted to say the following, we know you are out there and we know what you can do, and we are very impressed

Happy holiday weekend everyone!

Tomorrow is the fourth of July. This was the day The Declaration was approved and signed.  The day the Colonial Experiment became the American Dream.  The day everything changed.

This particular July 4th, is especially interesting.  I do not remember the last July 4th during a presidential campaign year because I didn’t happen to pay particular attention.  But today, our two (main) candidates will go on television and make flowery speeches and promise the world to us, just as long as we are willing to vote for them.  And then the political analysts are going to dissect and analyze every word and gesture and try to hide behind a journalistic lack of bias while steering the conversation in a particular direction.  And then the public is going to have private conversations about how a particular candidate is bad for our country and how the other guy/gal is going to be our savior.

Today’s political process seems very dirty.  The candidates appear inadequate, each in their own way.  The political commentators are biased and dishonest.  And the public seems to be the exact uneducated, unwashed masses, that are looked down upon by people who consider themselves the ‘elite’ and call the space between New York and Los Angeles, “flyover country”.

Somehow, I get the feeling that not much was different a hundred years ago.  Some Presidents have been unworthy of the title.  Newspaper publishers have been cheaters and liars, and the public, well the public literally didn’t bathe much.

But this is just an illusion.  The candidates at this level of the game are really the best of what is available to fill the Presidential shoes.  They may be unable to spell ‘potato’ or hold the book upsde down, but nobody can possibly stand up to this kind of scrutiny and constant pressure.  The candidates are good people who have already shown themselves to be good leaders.  The journalists are definitely opinionated but they maintain as much neutrality as possible and are balanced out by their colleagues on the other side of the political spectrum.  And the people are more politically savvy than they are given credit for.  When it comes down to it the system is brilliant.  And with all this imperfection, we somehow overcome the shortcomings and grow past the mistakes of our leaders.  Somehow, our country has been the innovator in almost every scientific field.  Somehow we have one of the highest qualities of life, and own the best ’stuff’.  Even with the bad economy, most people have an excess of food, a flat screen television, relatively new car, a cell phone and computer.  We have it so good that we are STILL selling SUV’s the size of Caribbean inner city buses.  We invented the Internet.  We invented the automobile.  We invented the telephone.

Whoever wins the upcoming election, things are not bad now and they are probably going to remain pretty good.  The reason for this, in my opinion, is the very nature of our society.  We are Open Source.  We have checks and balances, we have the right to criticize our leaders, to replace them if necessary.  The right to become candidates and fix the system from the inside.  The system is not perfect, but we are always working on it.  We are a society that is truly governed by the people, from the people and for the people.

On behalf of the eBoundHost team, we wish a happy 4th of July to the world.

About servers. Everyone reading this post is making a connection to a server. In fact, you are making a connection to at least a couple. There is a server in your office or home that is allowing you to proxy onto the internet, most likely a wireless router, which connects through another server, the DSL or cable modem. There is a caching DNS resolver server on your ISP. An entire army of router servers between your home and our data center. And the last server in the chain is our web server, which actually hosts this content.

Lets narrow down the definition of a Server. We are not going to talk about IBM mainframes or Sun UltraSPARC based blade systems. Today, we care only about the servers which comprise the majority of the infrastructure of the websites you visit. These are normal computers just like you have in your house or office, with the exception of being confined into more efficient packaging. They use familiar Intel or AMD processors, normal DDR2 or faster RAM, and SATA hard drives. What really separates them from home PC’s is the software. But software is not what this blog is about.

Here is what one of our older servers looks like (below):

To compare, here is one of eBoundHost’s newest servers.  This form factor is unofficially called the ‘pizza box’ due to its small dimensions.

The first thing you will notice is that the new server is not as tall. Our older hardware uses 2u (units of space) while the new servers use 1u. This allows for greater density. Some servers use as much as 7u but these are specialty machines that are filled to the brim with hard drives in gigantic RAID arrays.

Side to side comparison:

These servers fit into specialty (read expensive) racks that have 42u of storage in each rack. This means that when filled with 2u servers, we can only install 21 machines instead of 42 1u servers. It’s a dramatic difference when you talk about a server room full of racks such as in our facility:

Of course the entire 42 units are not available for servers, there are switches, power distribution units, firewalls, intrusion detection equipment.  All considered, we are happy to have 30 servers in one rack.

There is also the consideration of electricity and heat. A rack full of servers eats electricity like a hungry SUV, and produces just as much heat pollution. 30 servers stacked on top of each other, blowing air into the same direction, require an amazing amount of cooling, which needs big air conditioners that move a lot of tonnes of air. That’s all I’m going to say about that. Data center challenges is going to be saved for another blog entry.

To jump back into server hardware. Here is the same 1u server without its cover.

Motherboard, CPU, heatsink, RAM, hard drive and a very powerful cooling fan. Seems simple enough. Another picture:

Every server is custom built. When an older machine comes off line, we generally sell it through eBay and build a new server to take its place. The nature of hardware is such that components wear out and fail eventually. Our clients and our reputation are far too important, so we give old hardware the boot and use all new equipment.

Here are some servers in action. The following pictures may not be completely safe for geeks, they may cause weakening of the knees and a desire to run out and fix something. Please refrain, it will pass:

These (above) are dedicated servers. Inventory tags have been obfuscated in order to protect the innocent.

(below) Are some specialty machines which have 15k SAS (fast/expensive) hot swappable hard drives in RAID array. Used for our shared servers, VPS machines, and some powerful dedicated servers.

Each server is built by our staff. We love them so much that we have hundreds of them

More to follow, there is so much to cover: data center, operating systems, server software.