One Tuesday night a few weeks ago, a janitor vacuuming in our computer room accidentally unplugged a cable, cutting off power to three of our firm’s servers and damaging some hardware.
Coincidentally, the same night, a power outage at Intuit knocked out that company’s primary and backup website systems. Approximately 300,000 businesses that use Web-based products such as Intuit Online Payroll, Quicken Online, QuickBooks Online, TurboTax and QuickBase were left without service.
Jeffrey Joseph, who took over from me awhile back as Palisades Hudson’s main technology troubleshooter, discovered our problem around 11 p.m. when he tried to sign onto our network from home to do some overnight maintenance. Jeffrey came to the office and worked through the night. By the next morning, he had everything up and running on a replacement server. It was business as usual when our client service staff arrived for work.
That night, when our staff was heading home after a productive day, Intuit was still working to get its systems back on line. “Our business has been 100% on hold for all of today,” one Intuit customer said at around 3 p.m. on Wednesday. Another customer said in a comment addressed to Intuit, “I run my entire business off of your online service. With that said, I can not write checks of any type and/or post information to my books at all. This is truly ridiculous.” Some businesses that use Intuit to process credit and debit cards reportedly could not make sales, which in many cases were probably lost to competitors.
Intuit scrambled to get its systems back to normal. A message that appeared on the QuickBooks support page said, “We are committing every necessary person and resource to restore the important services our customers rely upon as quickly as possible.”
But regardless of Intel’s determination to fix the problem, our employees were able to get their work done that Wednesday, while many of Intel’s customers were not. The systems that are critical to our company’s day-to-day work are directly under our control, which puts us in charge of our own data processing destiny. Small businesses that rely on Web-based services may have access to cutting-edge technology and 24x7 service, but they ultimately place their fate in someone else’s hands. That is the danger of computing “in the cloud.”
At Palisades Hudson, we use remote data systems for several purposes. We keep our company calendar on Yahoo, for example, which is easy for all of our employees to access from virtually anywhere. But this is not a critical application. Our clients’ tax returns and investment statements will go out, and our employees and vendors will be paid on time, regardless of what happens to any off-site computers.
Still, many small businesses, especially those whose owners have too little capital to keep spare hardware available or too little technical skill to manage the requisite hardware and software, are better off outsourcing their technology functions, even at the risk of an occasional outage like Intuit’s.
No system is bulletproof. The best backup system in the world will fail if, for example, management does not make timely backups, or if a single accident can disrupt both the primary and backup systems. Intuit certainly had backup systems, but they were of little comfort to customers whose operations were disrupted for 36 hours.
Our firm’s self-sufficient culture grew out of the fact that I had a decent working knowledge of technology when I first hung my shingle outside my one-man office. I was willing to devote time to technical issues, to train people to help keep things running, and to hire outside help — on a strict budget that I had the skills to manage — when I wanted to upgrade out capabilities.
Preparing for occasional disasters requires a lot of strategic thinking. Now that our firm has expanded beyond Scarsdale, N.Y., we are planning an upgrade that will allow us to maintain a second set of data facilities at our Atlanta office. This will make it easier to keep everything running in case of a power failure or natural disaster in Scarsdale. But it is a technically challenging move, since we must maintain identical copies of our data in both locations, and provide a way for users in all our offices to log into each set of servers as needed.
Yet avoiding disaster is also sometimes a matter of luck, and even a near-miss provides opportunities to learn from what could have happened. Our janitor, for example, should never have been in our computer room. That message was quickly passed along, and we will probably follow up by installing a new lock on that door.
And Jeffrey, who is the hero in our story, does not log onto the network from home every night. Even if he did, he still must sleep, and disasters can happen at any time. We need to look into systems that will automatically alert him and other managers, including me, when a problem arises.
Something will eventually go wrong, but something always does. Breakdowns and failures are inevitable. It pays to be prepared, and on a day like that Wednesday, it feels really good to be in control of our own fate.