Just a moment...

All posts by artur

Massive Denial of Service Attack

Hosting

ddos_attack

Here are the facts, our network was hit with a series of massive DDoS attacks this weekend.  More accurately it was one continuing attack that caused customer impact four times:

  • Friday, May 8th, 12:55 am to 1:30 am
  • Friday, May 8th, 2:50 pm to 3:15 pm
  • Saturday, May 9th, 11:20 pm to 11:40 pm
  • Sunday, May 10th, 1:45 pm to 2:45 pm

Each instance of the attack was caused by inbound flood of traffic aimed at one of our customers.  This resulted in dropped packets and lack of service to several network segments.

The customer that was attacked already knows that they were the target.  If you did not hear from us over the weekend, you were not the intended victim, just an innocent bystander.

Each attack was slightly different and our network team worked quickly as possible to bring the flood under control.  At one point over 10,000 network hosts were involved, so it took some doing to gain the upper hand.

DDoS is inescapable, it will continue to happen as long as there are bad people out there looking to take out a rival, whether another business or someone broadcasting a point of view they disagree with.  Searching for Denial of Service you will find thousands of recent examples of companies small and big reporting that they were targeted by such an attack. All major name brand companies, banks, telecoms and service providers have been impacted.  This is not avoidable because someone somewhere has the will and the means to break through any countermeasure.  Recently, even a DDoS remediation service was overwhelmed by an attack which was hundreds of gigabits in size.

However, every problem is also an opportunity.  Our current DDoS remediation is being adapted to meet the challenges of the evolving threat.  We already began the process of expanding our network and putting more automated countermeasures to ensure that the DDoS ends before a single person is notified.  We’re going to fight computers with computers.

Lastly, we are going to improve our communication during such events.  There was a lag between the service impacting event and the notification to our end users.  Without proper communication the end users only see a network slowdown or outage without the awareness that we are working nonstop to fix it.  So the perception of what we are doing does not match the urgency and seriousness of how we are treating the situation and this can add frustration to an already emotionally charged situation.  Any future service disruptions will be immediately reported to our social media feeds and timely updates will be pushed out.

These new procedures and hardware will take some time to integrate into our systems and workflow, but the processes have already begun and will result in better quality of service almost immediately.  We thank you for being an eBoundHost customer and look forward to hearing your feedback.

Relief effort for tornado victims in Washington, Illinois

Misc

Days after the colossal tragedy in the Philippines where Typhoon Haiyan caused deaths of upwards of ten thousand people, the midwestern part of the United States was hit by powerful tornados.

Just 151 miles away from the eBoundHost.com home base near Chicago, Illinois, a massive tornado touched down in the city of Washington.

The images of devastation and news of loss of life are so completely overwhelming that it’s amazing how anyone survived. A good portion of the city has been reduced to rubble, houses, businesses, stores, hospitals, schools are simply gone. People who planned their lives and retirements, and worked hard to build and to save, are left with nothing but the clothes on their backs.

The realization sinks in that this could have been us, any of us.

Most importantly it’s time to help our neighbors with the very basic needs in life like clean water, food and shelter. We are entering the coldest part of the year where temperatures are dropping to below freezing and these people need help today, right now.

Please join us in doing what you can. Volunteer your time or help by donating funds to the relief fund. More information can be found here:

http://ci.washington.il.us/

Hurricane Yolanda/Haiyan in Philippines

Misc

typhoon.yolanda.haiyan

Our hearts go out to the victims of Typhoon Yolanda/Haiyan in the Philippines.   We are extremely saddened by the loss of life and hope that the  survivors find comfort in the world community joined together to rebuild what was destroyed.  eBoundHost.com is donating to the relief fund  organized here:

http://www.itmatters.com/typhoon-yolanda/

100% of donations collected go directly to the victims. Wecall on everyone who can to join us.

http://www.youtube.com/channel/UC9s3PmasGVnw9Yl58aAuwiQ/videos

Faster hosting == more conversions and better SEO

Hosting, How it works

Sophisticated clients love to chat with us about search engine optimization (SEO), keyword placement, guest blogging, pay per click (PPC) and other techniques to increase the traffic to their websites and ultimately add to their bottom line.  They spend hours each month tracking changes in traffic patterns and spend thousands of dollars on advertising. They optimize the smallest details and change landing pages to rework traffic flow. But often, their results plateau and they don’t know where to look for the next step.

An important factor for search engine rank is site speed.  Search engines pay close attention to the speed of your website, because faster websites operated by more sophisticated webmasters are typically more important.   In fact the search engines consider it so important they give you tools to measure and help make your site faster:

https://developers.google.com/speed/pagespeed/
http://developer.yahoo.com/yslow/

More importantly, faster websites convert better. People don’t like to wait, even if the difference is barely perceptible.  You’re competing  against the likes of Amazon who has an unlimited budget and teams of experts watching for slowdowns.  A 2-4 second lag on each page, makes your business look like it’s being run by amateurs.  Often, repeat business is based on a “feeling” rather than actual product quality.  So you want customers who are just as impressed by the speed of your website as the quality of your product and customer service.

Getting rid of perceptible slowdowns is so important that even your iPhone or Android have built-in transitions such as swiping, sliding, zooming effects, spinning circles and “loading” progress bars. This eye candy is not just pretty, it’s there to help pass the time between launching applications.

Smart websites are built the same way, everything is optimized to reduce perceptible wait time and help remove any obstacles from the visitor to complete their purchase.  By the way, perceptible and actual load times are not the same thing.  It’s possible to have your pages start to render in the browser before the entire content has been downloaded from the server.

Website speed typically has more to do with the structure of the website rather then the size of pages or images. For instance, a page that consists of large blocking files which must be downloaded before the website starts to render, is going to be perceptually slower than one which places the blockers farther down the chain. The first option is to rearrange these files to load only when needed and move all external includes as far down the chain as possible. Google Analytics, for instance, should be loaded at the very bottom of the page, below all static content such as images.

However, this is not always an option because most websites today use a content management system (CMS) such as WordPress, Drupal, Joomla or Magento, which come with a prebuilt structure that cannot be rearranged. Additionally, plugins and themes for these systems have dependencies that require a certain structure.

But there are other easy things that can be done to dramatically improve the speed of delivery.

slow website
Original, pre-optimization

In the above example, each line item is an individual file. This is the breakdown of how long it takes to download each file. We are mostly interested in the top 10 entries’ blue area. As you can see the very first files are javascript (navigation) and css (design). These are large text files which block rendering and collectively soak up a full second of customer’s time.

The lowest hanging fruit in this case is to compress large text files in flight between the server and the visitor’s browser. Text files such as .js and .css, compress easily and are the biggest bang for your buck. Such a file may range between 50 and 500 KB but will compress 50-80%. When taken in aggregate for all your css, js and html/php files, it’s a very significant time savings.  Today’s servers have such an overabundance of hardware resources that the additional strain on CPU and RAM is not noticeable.  The trick is that it takes less time for the server to compress, transfer and have your browser decompress, than to simply transfer an uncompressed file.  Besides, this functionality has been built into browsers for a long time and it’s silly not to take advantage of it.

While the server and browser already support this functionality, it typically needs to be enabled explicitly on the server.  However, this needs to be done the correct way or it can backfire. Trying to compress the wrong type of file has serious penalties, so be careful.

Fast website optimization
Optimized, much much faster

This is the same website after server side optimizations have been applied. Notice how the blue portion of the top 10 lines have disappeared completely or have become significantly smaller.

Furthermore, the rest of the website files are downloaded much sooner and the website takes only 1.5 seconds to begin rendering instead of 2.6 seconds which is nearly twice as fast.

The takeaway here is the extreme reduction in transfer time is easily accomplished even without structural changes to the website.  It only has to be done once and now we can sit back and enjoy happier visitors and some very impressed search engines.

Denial of Service: 07/31/2013

Hosting, How it works,

This afternoon we experienced a massive distributed denial of service attack against the eBoundHost network.

Although we have been working to mitigate a DDoS starting at Noon CST (GMT -6) without customer impact, the attack escalated and began to impact customers at 1 PM CST. The attack was fully resolved at 1:52 PM CST.

Typically these events are handled seamlessly in the background without impacting our end users. However, the scale of this event was unprecedented and the skill of the attackers was considerable. Our network engineers were able to resolve this problem and have the impacted network segments back online within 55 minutes.

As always, we will use this incident as a learning opportunity to see how to adapt to the evolving attack vectors and protect our customers from downtime in the future.

If you have questions about any aspects of this attack, please reach out to our support team. We are here for you.

Regards,

Artur
eBoundHost

Godaddy’s Outage

Hosting

On 9/10/2012 Godaddy, the popular web hosting, and domain registrar became victim to online hacking or Denial of Service attack (DoS). The number of impacted websites was reported to be in the millions. It was without a doubt a difficult day for many of the small businesses hosted on Godaddy.

The infamous hacker group “Anonymous” claimed responsibility, but at this time it is unclear if it was a DoS, hack, or technical failure. Reasons for this attack on Godaddy at the moment are also unclear.

We would like to express our heartfelt sympathies for anyone who has fallen victim to these attacks. Many don’t consider the impact on everyday people, even though Godaddy was the one targeted.  The real casualties of this attack were as always, hardworking small business people who had no idea about Anonymous or their crusade.

Thunderbirds Last Flight

Hosting, , ,

Earlier this week, the Mozilla Foundation put out an announcement that distills to this:

To be more specific, Mozilla will no longer focus on developing innovations for Thunderbird but will keep it safe and stable … Mozilla will also provide all the infrastructure required for new, community-developed features to be integrated in upcoming Thunderbird releases.

In a nutshell, they are announcing the retirement of Thunderbird. Citing the reason of the project not being their “top priority,” which, in plain English means they lost the war to Gmail & Co on the mail side and they are going to focus like a laser so the same thing won’t happen on the browser. As of June 2012, Firefox holds 30% less market share than Google Chrome and is losing ground quickly.

Contrary to popular belief, open source software is just as expensive to develop as any commercial product, but the costs come in different ways. The same, professional high quality programmers, spend weeks and months contributing their code to a versioning repository where it’s reviewed, QA’d and then released. This is the same intense and expensive process as followed by Microsoft, Google, Apple and any other large development house. Sometimes these coders are working for a corporation with their full time being devoted to a single project, sometimes they are working out of their basement as unpaid volunteers. The Mozilla Foundation is no longer participating in this effort which means the open source world just lost an important ally. Thunderbird has been under the Mozilla umbrella since inception and it’s not likely that the OSS community will keep it going. Of course bug fixes and security patches will be available for some time but the writing on the wall is clear.

Which is pretty sad for users like us who are not using gmail or webmail interface and require a fast and flexible IMAP application to plow through a 10GB mailbox at local application speed, keyboard shortcuts and unrivaled ease of use.

However, time moves on and we need to look ahead. The IT landscape is littered with the remains of indispensable applications like ACT!, Eudora, Google Wave and others that had a sizable following at one time.

We’re currently considering other options like Mac Mail and Outlook Express to recommend to our customers, and we’d love to hear what you’re using.

Stop SOPA / PIPA

Misc

Millions of Americans oppose SOPA and PIPA because these bills would censor the Internet and slow economic growth in the U.S.

SOPA / PIPA

Two bills before Congress, known as the Protect IP Act (PIPA) in the Senate and the Stop Online Piracy Act (SOPA) in the House, would censor the Web and impose harmful regulations on American business. Millions of Internet users and entrepreneurs already oppose SOPA and PIPA.

The Senate will begin voting on January 24th. Please let them know how you feel. Sign this petition urging Congress to vote NO on PIPA and SOPA before it is too late.

https://www.google.com/landing/takeaction/

 

Primary link soft failure

Hosting

Thursday, September 16th 10:25 GMT -6

Ongoing network outage impacting a significant portion of the eBoundHost network has been traced to one of our main peering points. The peering provider has a “soft” failure and as such was not demoted from “preferred” status.

The link was not completely down since the outage was behind the next peering point and the router had to be failed over by a network technician rather than the router itself.

The regular secondary failover peer uses a diverse network which was apparently using the same provider as their primary and is now in process of switching them off. We estimate this outage to last another 10-15 minutes at the most.

This kind of network failure/recovery happens quite often, our systems are designed to automatically fail over. Such outage have impacted customers only 3 times in the last 10 years. We do not expect this to happen again any time soon but just in case, the primary network provider will be kept down until we can verify that everything works.

Quick update

Misc

It’s been quite a while since the last official blog post, specifically almost 6 months.  This is in no way due to a slowdown of eBoundHost activity, quite to the contrary, we have been so busy that it’s been difficult to keep up with non-critical things like this blog.

The last 6 months have brought a lot of behind the scenes changes to our team, office, data center, even a refresh of our office computers.

Most importantly we have added some fresh new faces to our customer support team.  Of course, the term “fresh” is relative, since our team members are industry veterans (if you can say that for a 10 year industry).  Those of you who have the pleasure to work with our support team will notice some new names on your support tickets, be sure to put them through their paces!  No need to go easy on them just because they are new.

Our data center has been expanded in order to accommodate projected growth.  This has involved new rackspace, more bandwidth and more electrical power.    Since even the smallest changes to the data center can lead to unexpected consequences, we try to plan ahead as far as reasonable and make incremental changes. The latest update should take us through the middle of 2011 without any more major upgrades.

And finally, we are in the midst of a office move.  We outgrew the previous office of 7 years and have moved up to a beautiful new office with large windows.  It may be strange to mention that this office has large windows, but you would be surprised at how many potential offices we turned down due to the lack of sunlight.  We might be computer people, but aesthetics and comfort of the office environment are paramount.  We spend more time working than any other activity, even sleeping takes less time during the week, so the office environment has to be comfortable.

That’s it for now, lots of other things going on that we can’t get into just yet, but a few major changes are in the pipeline.  Stay tuned!

Thanksgiving

Misc

Today is the American national holiday of Thanksgiving. We gather from all over the country and sometimes the world for the purpose of sitting down around the same table and sharing a meal with our closest family members. The news reported that 37 million people are traveling the highways on this day to be with their families. That is more than 10% of the country’s population on the roads at the same time trying to join their relatives for dinner. Dwell on the enormity of that for a moment.

In the hustle of life its easy to forget that this is really what is most important, family. We work to live not the other way around.

From our table to yours, we wish everyone a happy Thanksgiving!

Comcast “blinks” and we get calls!

Maintenance

A few minutes ago Comcast went down.  Boy, did we start receiving calls from worried clients who thought we were at fault!  Everything looks good on our side folks!  Nothing we can do to help until Comcast gets their network back up.

You really don’t appreciate a good ISP until you realize how rarely they have issues.

Call center phone problems

Misc

Technological advancements are not without caveats.  For instance, our VOIP provider is having issues and as a result our primary telephone line is not working for some callers.  Luck would have it that all of our other (secondary) lines are working without any issues, while the only one that truly matters is down.

For the next several days call this secondary number: 847-368-9490

We have used this particular telephone company for the past 2 years without any major issues and this is the first time we have seen how they respond to a real customer service trouble ticket.

Working with their first and second level support really makes me appreciate our own support team.  Our people know what they are doing and they don’t have idiotic scripts to read while you are aggravated on the other end of the phone call.  We don’t waste your time with endless “for this department press this, for that department press that”.  We don’t have you wait on hold endlessly while reading basic notes.  We don’t transfer you to five departments before reaching the correct person.  Bad customer service is infuriating.

If we don’t have a resolution tomorrow, we will move off to another carrier, telephone support is too important for our customers.

You don’t appreciate good service until you need it.

The joy of (more) speed!

Hosting

After blowing through deadline after deadline, our new bandwidth carrier delivered our newest ultra fast backbone connection.  What they lack in intra-department coordination they more than make up for with the quality of their bandwidth.  The new connection has great ping times from around the world.

  • Chicago:   12 ms
  • Stanford University: 32 ms
  • Czech Republic: 141 ms
  • Italy: 141 ms
  • Sweden:  128 ms

Our customers should see a decent response time improvement as well as raw download speed improvement.

Happy browsing everyone!

Network monitoring

How it works

After a conversation with a concerned customer we need to clear up the network graph situation.  The graph located here, is showing uneven traffic with gaps and dips and seems to drop down to near “0”.

We are absolutely NOT experiencing an outage or interruption.  We are just pushing so much traffic that we are hitting a limitation of our monitoring software.  When it reaches a peak our software becomes confused and  “wraps” around to start at zero.  The graphing software assumes that nobody could possibly push this much traffic, so it thinks the data is invalid.

We are looking for other graphing software options right now and should have a fix shortly.

Happy holidays everyone!

Misc

Whatever you celebrate, we can all join together and wish each other a happy holiday season.  With such awful weather (in Chicago), it’s the perfect time to gather indoors and spend time with the family.

Due to holidays our team is operating on a skeleton crew for the next few days but we are still here for you if you have any questions, concerns, or just want to say “hello!”

Have a happy and safe holiday season and we’re looking forward to working with you in 2009!




Just a moment...
Just a moment...