Tangled Webs

    A Meeting by the River:
The Mizuho Debacle
Issue 7.3
Jun 12, 2002

An Ocean of Ink

On April 1, three of Japan's largest banks began joint operations under the name Mizuho. This was the culmination of almost three years of corporate integration and resulted in the largest bank in Japan; by some measures, the largest in the world.

The computer systems-integration project lasted two years, cost a fortune and was a disaster. Millions of transactions were delayed. Thousands of people could not receive their salaries, pay their rent or withdraw their savings. However, few in the industry were terribly surprised by this. A few months before, similar but less severe problems plagued the newly merged UFJ's banking system.

It quickly became public that Mizuho top brass knew of the computer problems, but opened for business anyway. The executives had hoped that an army of office clerks processing transactions manually could keep operations limping along while programmers worked around the clock and patched problems as they arose. Obviously, they were wrong.

It has become fashionable for journalists and politicians to express outrage over such poor decision making, and several senior bank executives have resigned to take responsibility. Ex post facto criticism is always easy, but the real problem is that this kind irresponsible project management is common and accepted throughout Japan's financial industry. Unsurprisingly, today outraged voices were silent when this exact "launch and patch" strategy saved Japanese banks billions of dollars in 2YK rectification.

The Mizuho fiasco illustrates not so much the poor judgement of a a few executives, but a deep-rooted virus infecting Japan's software industry.

Go with the Flow

Mizuho's integration project was tough by any standard. Each of the three banks' systems were developed over a period of decades by different external contractors. Each system was likely quite different in design and implementation. Add to this intense political infighting among both corporate officers and software contractors, and you have a fertile field from which to cultivate excuses.

Such excuses, however, miss the underlying problem. Japanese banking software, in fact Japanese software in general, is of shockingly poor quality. Documentation tends to be copious, but irrelevant and incorrect, and there is usually no record of who designed, developed or modified various system components. In short, no one really understands how these systems work.

Change tends to come very slow in banking. Banks take their time tweaking and patching existing software until it somehow produces the desired output. Bugs are fixed as they are discovered, and a huge amount of data processing always ends up being done by hand. Over the decades in which these banks' software had been developed, it had undoubtedly evolved into a incomprehensible tangle of undocumented minor adjustments and patches made by hundreds of anonymous programmers.

Even this situation, however, is no excuse for failure. After all, Mizuho had almost three years to integrate their systems. The failure was a result, not of an insurmountable task, but of the fact that Mizuho, like the rest of corporate Japan, finds the current abysmal state of software engineering acceptable, and ran this integration project like all others.

Drink Deeply or Taste Not

Japan, more than any other country exemplifies engineering excellence. Companies such as Toyota, Nikon, and Mitsubishi hold themselves to such ridiculously high standards of quality that Westerners make jokes about it. So what happened to Japan's software industry?

Frankly, I could write several volumes on the subject and barely scratch the surface. The apathy towards software quality is endemic and spans the entire value chain; from the the production coder up to presidents of software companies and right out to the end customer. People don't really care, or perhaps don't know any better.

I believe that the problem stems from the fact that software development is not engineering. Granted, there are thousands of 24-year-old "Senior Software Engineers" who will tell you otherwise, but commercial software engineering only superficially resembles fields like automotive, mechanical, or civil engineering. Interestingly, Japanese firms excel at these disciplines.

Engineering, involves bodies of accepted knowledge and standard processes. Engineering has ethical codes, and cutting corners is considered unacceptable; often illegal. Engineering processes involve strict accountability, and those who err are quickly brought to task. Most significant perhaps, in engineering, quality is measured by clearly-defined, widely-accepted and understood criteria.

In contrast, software development embodies an utter lack of accountability. Cutting corners is encouraged at almost all stages, and quality is widely considered unmeasurable or even irrelevant. This dismal situation is not unique to Japan by any means, but it has affected her far worse than other nations.

In conclusion, I would like to voice an extremely unpopular opinion. Mizuho's decision to go live was the correct one, and I think most people in the industry know it. Given the state of Japanese software engineering, Mizuho could have spent two more years integrating their systems and they would have very likely experienced similar problems.

Mizuho gambled and lost, but they had no other option available to them, and the fact that they had no other viable options is far more shocking and significant than Mizuho's temporary computer problems.

[ Home Page] [ Back to Index ] [ Previous Issue ] [ Next Issue ]

© Copyright 2002, Tim Romero, t3@t3.org
This article first appeared in the June 12, 2002 edition of The Japan Times.
Tangled Webs may be distributed freely provided this copyright notice is included.
The Tangled Webs Archive is located at http://www.t3.org/tangledwebs/index.shtml