The Gomez Performance Index - Year In Review

Looking back on 2002, Web site performance played a significant role in how financial-services firms evaluated the effectiveness of their online offerings. Many of the leading players realize that fast site response times and strong availability is paramount to maintaining the indispensable channel that the Internet has become.

While trading volumes during the year remained well below historic highs reached pre-April 2000, brokerage firms did an exemplary job of maintaining superior site performance. However, some firms struggled to keep pace with demand during high traffic periods, suggesting that they may need to examine their load balancing strategies to keep site availability at peak efficiency.

Savvy business and technology stakeholders regard performance as a common goal to achieve a higher level of customer satisfaction. In fact, in many instances brokerage firms compensate key IT managers on their contributions to meeting and exceeding site performance standards.

The Gomez Performance Index (GPI) has tracked Web site speed and transaction success rates at online discount brokerage firms for more than a year. Our current transaction (published bi-monthly in Wall Street & Technology) does the following:
-Accesses each firm's homepage
-Enters the site using a secure username and password
-Requests a full quote for a specific security
-Initiates a buy order
-Previews the order
-Cancels the order and logs out
The goal of this transaction is not to measure the speed of order execution across online discount brokerage firms. Instead, we measure the amount of time necessary for the Web site to load each page and process the information necessary to facilitate this process.

GPI tests are initiated every 15 minutes from multiple co-location facilities across the U.S. at T1 Internet connection speeds. These tests are then aggregated to formulate benchmarks, representing the average time to complete the task without any user latency. Additionally, we calculate the percentage of successful tests without encountering any fatal errors. These metrics provide an accurate indication of how each site performs from day to day, providing insight into performance across the industry as a whole.

Quantifying, Qualifying Performance
Response time performance for the discount brokerage transaction averaged 12.03 seconds across the nine firms on the GPI for brokerage since May 2002, the month we began publishing the index. Index highs reached 12.9 seconds in November, which is two seconds slower than the fastest monthly Index average of 10.9 seconds posted in October. Ameritrade registered the fastest response time in every Brokerage GPI published, save one. The firm executed our transaction in just over 5 seconds on average. Charles Schwab was the slowest overall response time performer during 2002, with an average task time of 16.5 seconds.

Yet, performance is a relative component in the overall evaluation of online customer experience. The contrasting business models of Ameritrade and Schwab make that clear. So to provide perspective on our regularly published performance metrics, Gomez also evaluates content, tools and functionality offered at each site. Not surprisingly, sites offering the fastest online offerings typically do not provide in-depth information, comprehensive visuals or compliance checks that are likely to impede site performance.

Aggressive site updates (i.e., feature and functionality additions) weren't as in vogue last year as they had in previous years due to functionality saturation and budgetary constraints. Brokerage houses were more focused on sustaining their existing online presence and services rather than implementing new Web functionality -- with the exception of firms that reached out to highly active traders (e.g., Ameritrade and Fidelity). Thus, site performance becomes a critical component for firms seeking differentiation and in the development of customer acquisition and retention plans.

Seasonal Shifts
As expected, overall GPI performance shifts from month to month, ranging from the fastest response time of 10.9 seconds in October to 12.9 seconds in November. Additionally, site performance was generally faster during the late summer months and into the fall when transaction volumes are historically lighter.

Beyond performance and functionality metrics, Gomez also evaluates consumer traffic patterns, through data provided by ComScore, across firms (particularly those tracked by our performance indices) to determine the impact load has on performance. What we've found is that most brokerage sites were well equipped to handle traffic during peak trading hours, but that performance does suffer at some sites due to increased load.

Not surprisingly, the lowest traffic months of 2002 (August and September) coincide with two of the fastest performing months. For example, Fidelity's traffic during August and September averaged 1,336,000 unique visitors per month, with an average GPI task response time of 12.2 seconds. This compared favorably to November and December, when the site averaged 1,502,000 unique visitors per month and average response time was 15.4 seconds.

Beyond Site Crashes
Historically, Web site performance was an issue only when an outage occurred. With more resilient internal infrastructure and third-party service providers adhering to stringent Service Level Agreements (fueled by industry consolidation that has resulted in fewer carriers with more than enough bandwidth to spare), brokerage houses were able to provide Internet delivery in 2002 that by historical standards exceeded clients' tempered needs.

Benchmarking industry performance, however, reveals that subtle variances in speed and success rates are critical to the success of a Web offering. The bottom line: site performance may not always equate to business success, but it plays a large role in customer satisfaction and achieving Internet channel effectiveness.

John Lovett is senior performance analyst at Gomez, Inc., a Waltham, Mass- Internet benchmarking and advisory services firm. Vice president of research Dan Burke contributed to this analysis. John and Dan can be reached at [email protected] and [email protected], respectively. For more information on the Gomez Performance Index for brokerage, point your browser to https://www.gomez.com/main.aspx?m=5&s=3&x=2.