How ANALYSYS Survived Snowmageddon 2010

What you can do to ensure business continues in the face of a disruption to business as usual.

I’m so proud of my team at ANALYSYS.  We didn’t skip a beat during Snowmageddon 2010, and quite frankly, we shouldn’t have.  There’s a couple of reasons why.  First, we are expanding from a regional Washington-Baltimore (or Baltimore-Washington depending upon where you hail from) footprint into a national one.  We have national and multinational clients on the books that expect service continuity from us.  Second, the Computing as a Service (CaaS or Cloud Computing) and data center technologies we offer involve a strong resiliency message, and a blizzard of historic proportion is the perfect example of a disaster we say our solutions protect against.  Therefore, it would be hypocrisy if our team wasn’t able to get work done.  Third, we prepared in advance.  Long ago, we developed plans to accommodate unplanned circumstances such as power loss, theft, vandalism, and of course natural disasters… and we keep them up to date.

Now seems to be a perfect time to share some of these practices which made last week a success.  In my opinion, it’s a great case study on what any small business out there should be doing to protect themselves.  To put things in perspective, remember that this storm was so bad that nobody could get into work.  This meant all of us had to work from home and still meet our goal of transparency in the experience ANALYSYS clients received.

Secure Remote Access - Every day ANALYSYS is at work, we are operating remotely.  That’s right, when we are

ANALYSYS Data Center Operations in Baltimore

sitting at our desk on a normal day, the data we work with exists somewhere else – not in our office.  So, from a computing standpoint, even though our team worked from home, they didn’t need to change anything.  Fast?  There’s a local server (some file sharing, print, and Active Directory) in our office plus fast internet access to accommodate the demand and keep things speedy.  Secure?  From a physical standpoint, the data center in which we placed our systems is incredibly secure (SAS 70 Type II certified), more secure than things would ever be in our office.  From a networking standpoint, we use the same firewalls and security practices we would use if our systems were connected to the internet in our office.  Reliable?  There’s redundant internet access from our office in case there’s an outage, the data center has a multi-million dollar investment in battery, HVAC, generator technology.

What steps can small business DIYers take to emulate this type of solution?

• Partner with a data center and move your systems off-site.  Go take a tour of the facility prior to signing up.  Yes, that means doing business with someone local.  You don’t want to hop on a plane every time you need to your servers, nor do you want to pay a data center help desk technician $200 per hour every time you need hands on-site.
• Install connectivity, such as a point-to-point T1.  If your needs are modest you might also get by with a Verizon FIOS or Comcast connection.  Try to ensure you have at least two in case one fails.  Internet connectivity is your weakest link in the chain between users and their data.
• Implement the routing.  Check out SonicWALL’s SSL-VPN product.  It’s a simple, affordable solution for small businesses that want to do remote access themselves.
• If you want to institute desktop virtualization, take a look at Microsoft’s Small Business Server Remote Web Workplace, or for a slightly more robust solution, check out Citrix XenDesktop.

Voice Over IP Telephony (VOIP) – Much of the public believes that VOIP phone systems will operate anywhere.  That’s not necessarily the full truth.  VOIP by its very definition means a phone system that operates over an IP (computer) network.  However, depending upon the make/model of the phone system these may or may not offer functionality beyond one network segment (one physical location).  Furthermore, the way VOIP systems define the anywhere, or remote, capability, critically depends upon how it’s set up.  As for us, our phones work anywhere we take them, in the office or out of the office.  In order for someone to work from home, they simply pick up their phone, take it home, and plug it in to a connection with internet access.  As a backup, we can also have our phone system forward calls to different extensions, and even home phone numbers if necessary.

So, what do we look for in a VOIP solution?

• Before you begin your research or call anyone, define your expectations clearly.  Unlike computer networks and remote access VOIP requires involvement from a blend of service providers and equipment that is exceedingly complex and perpetually in a process of innovation.  Once you have your goals outlined, consider retaining the assistance of a qualified telephone system installer.
• Explore redundancy thoroughly.  Many phone lines are digital these days.  This adds a tremendous amount of potential but also adds complexity, where something as simple as a power outage or a cut wire to a box somewhere in the closet could mean an entire system outage.  Implement a second technology as a backup plan and ensure your systems automatically fail over.  Technologies to look at combining are traditional copper (aka Plain Old Telephone Service or POTS), SIP trunking, shared data/voice over a T1, cellular backup, and ISDN.
• Save money!  Companies that have not changed their phone systems in more than three years, may be able to reduce their recurring costs of phone service by switching to newer technology.  As opposed to contacting the telephone companies themselves, consider retaining the services of a consultant that specializes in this area.  You’ll have an expert in your corner who will advocate before the service providers that want your business.
• Monitor your systems for outages.  As opposed to waiting for someone on your team, or worse, a customer, to note a service problem, get out in front of a service outage with tools that send you an email, text, page, or telephone call when there’s a problem.  These tools are easy to use and come in the form of software or appliances you can own, or even can be purchased as a service from a third-party.  Ipswitch’s WhatsUp Gold or GFI’s Network Server Monitor are great tools for small businesses.  Solarwinds offers solutions for more demanding environments.

Battery Power & Generator Backup - Nothing is more frustrating than losing work from a momentary power outage.  A longer term outage can be crippling.  For how expensive a power outage can become, not to mention the unplanned impairment to team efficiency, batteries and generators are tremendously low cost and respond to the most frequent unplanned disaster businesses suffer.  So, although battery backup and generator systems didn’t play a role in our response to Snowmageddon, I felt that it was an important detail to include.

• Protect workstations.  A $40 per seat investment in a workstation battery backup enables our folks to save their work if the power goes out, renders blips irrelevant, and gives the backup generators time to come online.  We use APC BackUPS for this purpose.
• Protect infrastructure equipment.  Just like momentary coverage for desktops, if you want a fully functional network, you’ll need battery backups for everything else, including firewalls, routers, switches, wireless access points, etc.
• Install backup generators.  These aren’t expensive, and surprisingly many landlords welcome the improvement if the tenant is willing to contribute the capital.  Within the electrical contracting community, specialists exist who focus on implementing backup generators.  Contact one and have them help with this initiative.
• Go green.  Do you own your own property?  Great!  While they are installing your backup generator, save on your power bill by engaging your electrical contractor for help with improvements that reduce power consumption.  For a more modest step, consider implementing smart surge suppressors and power saving measures on your network that shut systems down automatically when not in use.  Does the landlord pay the power bill?  Beware – with the cost of energy spiraling upward, many rental agreements are now being renewed net of power costs.  So, how does this relate to backup power?  Remember, if you manage the technology that’s running to keep it to a minimum, that’s less chance you have of an overload.

Well-Defined & Communicated HR Practices – No amount of technology, however elegant, will deliver results if the team is not prepared beforehand.  There are four places in time that we look at.  First, there is strategy which is ongoing and evolutionary.  We want to ensure that our practices are relevant to the business we presently are.  Second, there is planning related to a specific disaster we can forecast.  A snowstorm is definitely a place where we can engage in substantial advance planning.  Third, there is management of the disaster as it happens.  Our goal is to remain prepared, flexing our planning to respond to circumstances which we did not foresee.  And, finally, there is a retrospective on the disaster, as a means of understanding how we can improve.  Substantial content could be devoted to this topic, making it a great opportunity for another post.  For now, I’ll share some of the key areas that helped us through Snowmageddon.

• Work at home policy.  Define in advance the expectations for work performed from the home.  Who will be authorized to perform work from the home?  If there will be only certain individuals allowed to work from home, how is the approval process applied, and applied fairly?  How are hours worked recorded?  What outputs are expected?  Will the company compensate for any costs such as telephone or internet access?
• Leave time approval.  How does the decision to implement a liberal leave policy or to close the company impact your employees pay?  Will employees be paid, or will they have the opportunity to use personal leave time?  Equally as important, how will team members register whether they are in or out for the day.  ANALYSYS has a inclement weather line which is updated with our decision to open or have a liberal leave day.  If team members can’t make it in, they register their status on our company intranet prior to the start of the day, making it easy for supervisors to plan accordingly.
• Points of contact and availability.  It’s easy to create a one size fits all policy that covers what to do in the case of inclement weather, where staff contact their supervisor for instructions.  What happens when members of the management team can’t make it in, but their departments must continue to function efficiently?  Resources must remain flexible and highly communicative.
• Be proactive!  Let’s imagine in this disaster that HR has a question.  With advisors such as HR consultants and legal counsel and government agencies closed for business, the questions end up going unanswered.

Advertisement