|
|
|||
|
|
|
|
|
|
Back to Facility Manager
Contents
|
|
|
|
It has been many years since I first paged through an issue of the NFPA magazine. As I looked through the pictures and articles detailing fires that had taken place, I was struck by the photos of blocked doors and empty fire extinguisher cabinets – avoidable and unfortunate mistakes. I would never want to have a fire in our theater for the obvious reasons of not wanting any lives threatened or any damage to the facility. Of course I knew those things already. What I promised myself while looking at those pages was that I never want to have the dubious honor of having pictures of the Herberger Theater Center featured in the magazine, knowing that people would be looking at them and thinking, “How could they have been so foolish and shortsighted?” While the Herberger Theater Center has not been featured in that magazine, nor has anything life-threatening taken place, something has happened that caused an interruption of our business that we are still working to recover from. If you consider that one of the benefits of the NFPA magazine is for people to learn what not to do, I offer to you our recent experience. Some of you may read this like a curious onlooker driving by an accident. Some of you may ask how we could have been so foolish and shortsighted. But I hope at least a few of you will take some time after you read this and make some changes within your own facility so as to learn from our mistakes. The day started like any other with staff filtering in throughout the facility. A brief conversation took place regarding our computer server and our plans for replacing it. Then the phone rang from downstairs because staff was unable to access the server. I tried it myself and suddenly the system stopped, my computer froze and major areas of our business halted. What happened was a total physical failure of our file server hard drive. Even a high tech company with specialists and a “clean room” were unable to recover anything. A separate functional hard drive stored our email but without the main server, using email was impossible. No problem, right? We just replace the hard drive, move the data from the backups and start the train rolling again after a short delay. That was certainly what we thought would happen if we had a failure prior to our imminent plans to replace the server. However, the reality was much different. We had a tape backup system that was running every night. We could have just grabbed the data from there. Right? Wrong. The tapes did not contain any recoverable files. We did not consider the tape system to be 100 percent reliable so we had another layer of backup. We assigned a specific staff member the weekly task of making CD data disks of all of the information from the server. Unfortunately, I soon discovered that some of the backup CD’s were several months old, some were more than nine months old and some information had never been backed up at all. When a new system was finally up and running after several painful weeks, we ended up with huge holes in our data. In our development area, all that was left was a partial list with names and addresses. In finance, the last 14 months were gone. In marketing, most of the recent projects, including the final draft of a facility brochure were gone including graphics and photos. The list went on… We are currently re-entering data from paper trails in both development and finance so we will eventually be whole again in those areas, but it will take months. The other departments are spending time re-creating forms and scouring old e-mail for files. All of this begs the question as to how this could have happened in the first place. It is definitely a “would-a, could-a, shoulda” story. The Herberger Theater Center is operated by a non-profit organization that exists in part because of in-kind and cash donations and grants from corporations and individuals. Like most non-profits, we are always tight on money and always trying to get the most out of what we have. When the IT company we work with informed us nine months earlier that the server was over three years old and that we should replace it and upgrade the software, we began a process that obviously took just a little too long. We sought a second and third opinion as to what we should do and painstakingly researched our existing system so that we could apply for grants to fund the project. Additionally, we examined our budget to see if we could find any money. Eventually, we were not able to secure any grants or outside funding for this critical need. With an eye on phasing the project, we came up with a plan for exactly how to replace the system and what features it would have. We were ready to move forward after the plan was reviewed by a board member’s organization, just days before the system crashed. As a small non-profit organization it is easy to skimp and get by with as little as possible. It was this thinking that cost us dearly. We should have been thinking, “pay now or pay later” because we ultimately paid much more on many different levels by allowing the system to crash instead of replacing it on a scheduled basis. So as you page through this magazine and maybe think to yourself, “How could they have let that happen?” I hope that you will make a commitment to look at your system and backups to make sure you are never in this situation, especially if you are a smaller non-profit organization trying to squeak by. People have said to me that a crash like this only happens to you once. I hope that my story will keep this from ever happening to you. Mark Mettes is vice president of operations for the Herberger Theater Center in Phoenix, Arizona. He may be contacted at mmettes@herbergertheater.org. |
||
|
©
2002-2005
International Association of Assembly Managers
|
