We're Out of Control!
If computers are becoming easer to use and more reliable every year, why do support costs per machine keep going up? In industry parlance, it's called "computer services." Most people, however, refer to it as "support" or more likely "keeping the damn system running."
Actually, keeping the damn system running is one of the fastest growing sections of the computer industry. Compaq's acquisition of Digital was, in part, motivated by their desire to get into the services industry, Borland, recently renamed Inprise, has restructured itself to focus almost entirely on the services market, and IBM's computer services division is now pulling in $19 billion a year. That's a lot of money to keep such reliable IBM systems running, and should raise obvious questions about conflicts of interests.
I noticed the prevalence of this conflict of interest a bit closer to home a few months ago when I was contracted to evaluate competing bids for the installation and support of a new LAN. It was clear that none of the bidders were making much money on the hardware, and one salesmen even stated outright that his company was willing to take a loss on the initial sale in order to win the support contract. As the soon-to-be IT manager nodded in approval at the bargain he was getting, I blurted out "But that means if your company sets up a reliable network, you'll loose money." I deduced from the awkward silence that followed that I had once again managed to offend everyone in the room. My point, however, remains quite valid.
Most network support arrangements are based on rewarding undesirable behavior. The more problems a network has, the more money the support company earns. From a client's point of view, an idea network is one that is stable. However, from a support company's point of view, an ideal network is one where there are enough problems to generate significant revenue, but not so many that the client begins to suspect duplicity or incompetence.
To be fair, I don't know of any Tokyo-based support companies that actually operate on the above model. However, steps do need to be taken to check or eliminate this inherent conflict of interest. One simply cannot assume a support company will act in such a way as to minimize its potential revenue. The following two steps will help ensure that you get the support you pay for.
Write it Down
Engineers hate documentation, but thorough documentation is essential in maintaining a stable network. When your system is set up, you should receive system documentation which, at a minimum, describes the hardware purchased, the software installed on each machine, and the significant configuration settings.
After installation is complete, the support provider should fully document every service call in a service log, and record any significant changes made to the network. There should be a clear record of what problems were found and what actions were taken to rectify them. This documentation, both system and service, should be kept in electronic form, preferably a database, so that it can be easily maintained and searched.
Lord knows documentation can be taken to extremes, and having non-technical managers decide what documentation is required of engineers is a proven recipe for frustration, resentment, and quite a few Dilbert cartoons. Service documentation should be kept brief. Documenting a three-hour service call should not take more than ten minutes.
Documentation serves three main purposes. First, it can greatly reduce the time it takes support engineers to identify and fix problems. Second, it makes it much easier to bring new engineers up to speed. Turnover among network support engineers in Tokyo is staggering. You cannot count on having the same person or group of people for an extended period of time. Third, documentation will help you evaluate the quality of service you are receiving from your support company and reduce the cost of changing providers if the need arises.
Unless you are an network engineer yourself, however, evaluating quality of service from service documentation is tricky business. It is important to understand that it is not at all uncommon for problems to take hours to locate and only minutes to fix. Furthermore, even the best engineers agree that networking involves a bit of voodoo. Sometimes a problem really can be solved without identifying the root cause simply by rebooting the server, reinstalling a driver, or reconnecting a cable.
Having these types of entries in the service documentation don't necessarily mean there is a problem. However, if you notice that the root causes of the problems never seem to be identified, or if the same problem keeps occurring with the same "solution" being used again and again, you will want to have a talk with your support company.
In my opinion, the way network support is billed needs to be fundamentally changed. Support companies should make their money by keeping networks running smoothly, not by constantly fixing unstable systems. This can be achieved by contracting for support at fixed per-machine per-month prices.
With one simple step, keeping the network as stable as possible becomes in both the client's and the provider's best interests. Every time your network goes down, your support company will loose money. The more stable your network, the more money your support company makes. And that is how it should be.
Since such an arrangement shifts all financial risks to the service provider, I expected strong industry resistance to the proposal. Happily, I was quite wrong. Most companies I spoke to were very open to the idea. A few were already experimenting with it, and all but one said that they would be willing to consider providing support at fixed monthly prices.
Obviously, there is much to be negotiated in such an arrangement. Most support companies will want to limit maintenance that could be done by the client, and evaluate all software for stability before it gets installed. If that unauthorized Hello Kitty screen-saver brings down your network (stranger things have happened) restoring it probably won't be covered by the monthly fee.
The fixed-fee arrangement is clearly in the client's best interest, but it is not for everyone. It requires that both client and provider be accountable for their actions, and that is often politically difficult. Particularly when dealing with Japanese client companies.
Furthermore, I have listened to far too many IT managers actually bragging about how many consultants it takes to keep their system running to assume this is a rational industry. When faced with a stable system that requires little maintenance, some managers inevitably begin to wonder why they are paying so much "for nothing." They count up the number of hours engineers spend on site and then shake their heads about how much they are being overcharged.
Rational managers will check that impulse quickly. A company contracted to keep a system running smoothly should not be penalized for doing so. Of course, not all managers are rational, and some really are happiest with systems that are unstable and require a cadre of engineers to keep them running.
It's no wonder network support is such a fast growing industry.