The application support teams had no formal way of measuring performance. They would log into the platform, try a few functions and time them on their watches. "It was a terrible way of troubleshooting," says Hirschauer. "We didn't have any good tools in place that could give us information about our transactions as they flowed across our systems." When a problem occurred at Wachovia, as at many large firms, finger-pointing would take place, each group insisting their systems and logs looked fine. "But there was obviously a problem somewhere and it was taking a considerable amount of time to resolve these issues," he says.

Wachovia sought a tool that could provide concrete metrics on the transaction flow from start to finish, so that IT could be alerted to and fix performance problems before hearing about them from customers. It had disparate tools for measuring the performance of operating systems, networks and applications, but those didn't tell the whole story, they only reported on one piece of the puzzle. Another tool, business service management software that created synthetic transactions to try to measure application performance, was also inadequate because it merely determined whether or not the application was up and running. "The real key for us was to know the overall timing for all of our transactions and then break that timing down into the multiple tiers our transactions were traversing, so that we could know, for instance, that we had a problem in our application tier or database tier," Hirschauer says.

Someone on staff had used OpTier's CoreFirst transaction management software at a previous job and Hirschauer's group decided to give it a try. Implementing the new software was simple for application servers and web servers for which OpTier provided out of the box integrations. But for homegrown and C++ applications, such as a complicated mortgage processing application, the process was more difficult and time-consuming, requiring code to be embedded in the applications. Yet Hirschauer says, "It's absolutely worth the effort in the end."

The software went live in August 2007 and began reporting on the performance of the foreign exchange trading application. At first the software just gathered metrics, basically on the amounts of time various functions took to complete, which were then shown to the developers supporting that app (who were not the original authors of the program) so they could fix any inefficient portions of the code. "Looking at the CoreFirst data, we were able to see that some of the coding practices used were poor and some recoding work needed to be done; it wasn't an infrastructure issue, it was a code issue," he says. Certain commonly used functions were running slow – one task had an average response time of 12.809 seconds. After the developers recoded the program, the average response time dropped to .968 seconds -- a 92% reduction.

Not all problems have turned out to be code-related, some of the results have shown infrastructure issues such as poor load balancing among servers.

The overall response time for the foreign exchange application has been reduced 33% and customers have noticed, Hirschauer says.