PM Essence

Setting up a Multi-Vendor Software Testing Cente of Excellence

- Dinesh N. Raisinghani, PMP, Capco Technologies


Most organizations would agree that if they are able to rapidly implement new software products/applications that are of a consistently high quality within minimum budgets, they would enjoy significant competitive advantage. In spite of this, challenges surrounding software quality remain untouched as testing is still perceived as a huge bottleneck. Organizations have spent millions on structuring their test organization but a large part still remains immature due to complex business and technology needs that takes priority over software testing. 

pma21

Businesses lately have adopted the Testing Centre of Excellence (TCoE) models to structure their test organization. It will be mostly internal to the business and have no participation from the vendor organizations that provide services to the businesses.

 

One of the practices is to set up a Multi-vendor Software Testing Centre of Excellence (MvTCoE) within the IT organization. Any organization dealing with multiple vendors can adopt and unravel benefits of cross project teams from different organizations. Key project management processes are required to be followed by the organization for implementation of MvTCoE. Additionally, the benefits pertaining to people, processes and technology are listed out for both the incumbent organization and to the vendor organizations that can be reaped by adopting such a model without compromising the intellectual properties of any participating organization. Let’s understand this model by a case study of a large Canadian bank that has gained the testing maturity by realizing the benefits of adopting this model.

 

Challenges faced by the business:

Due to business consolidation and inorganic growth over the years, the bank has grown vast resulting in their IT portfolio having several inter-related applications tied together in a complex architectural framework. The development and application management of these products are sourced to multiple vendors and IT service providers who provide development, testing and application management services. Testing of these applications had taken a toll on the IT organization within the business as it requires a lot of planning in terms of impacted application availability, network and infrastructure issues, down times. Though these challenges are perpetual, the time it took to plan the testing activities used to get prolonged unnecessarily due to lack of ownership and responsibility from associated application owners.

Risk due to multiple technologies

Chaos in implementation

Disruption in functional dependencies

Lack of testing knowledge management and inadequate test artefacts resulted in ad-hoc testing measures and sub-standard releases

 

Formation of Multi-Vendor Software Testing Centre of Excellence (MvTCoE)


To deal with the pressures by the operations team and delivery timelines associated with every project, the bank planned to implement a unified shared Test organization that would have participation and ownership from their strategic sourcing partners and key vendors.

 

The MvTCoE was planned to be a centralised, shared testing organization that the business establishes jointly with vendor organizations (strategic partners and other vendors) to enable a process-centric approach towards testing and enhanced the business value delivered by optimizing overall cost, time and effort. The CoE model revolved around key strategic assets of people, processes and technology with some guiding principles.

Every testing project executed within the organization was owned by this CoE.

 

Key vendors and service providers who provided application/product testing services were a member to the MvTCoE (there was a selection process during the CoE initiation phase).

 

A Test Management Office (TMO) within the incumbent organization was setup to provide governance and strategic insights to the projects.

 

Test Managers of respective projects from vendor organizations reported directly to the CoE lead. 

 

The ‘Projects’ shown indicate the projects and programs executed by the vendor organizations. With MvTCoE, these project owners jointly formed a unified and centralised testing group led by a CoE champion from the incumbent business (the IT org of the bank) and supported by the Test managers from participating vendor organizations.

 

Once it was decided to setup an MvTCoE, the bank carried out a step by step assessment and implementation as described in this picture.

A project management responsibility of MvTCoE includes Stakeholder management, Communication management, Scope management, Risk management, Resource management, Knowledge management.
pma21

The bank created a unified testing MvTCoE with participation from key vendors who supported critical applications. This CoE acted as a centralized test organization that employed quality assurance practices, plan and monitor the outcomes. Some of the key benefits that the business has realized after implementing the MvTCoE are:

 

Increased ownership from vendor organizations when it comes to managing application changes and release management.

 

Efficient resource utilization in cases where resources can be cross trained on other internal applications and can be utilized in a staff augmentation model.

 

Effective knowledge management by creating and owning a central source of project learnings, project planning.

 

TMO provides much needed governance and overall leadership that facilitates effective and efficient CoE functioning.

 

Seamless communication between different organizations supporting the business applications. 

 

Centralized management console for risk and compliance management.

 

Effective planning driven by the CoE optimized the test process, reduce effort while ensuring minimal cost and schedule variance

 

Unified test plan document that standardized testing levels, scope, deliverables and milestones across programs.

 

Solid change Management procedures helping application owners better manage the changes that impacted upstream and downstream systems.

 

Implemented common practices for defect logging, prioritizing, fixing, and tracking to closure. Defect leakage and production issues got significantly reduced.

 

Provided adequate guidelines on test data management, test environment and data management for every vendor to avail the common information.

 

Prior planning for network accelerators improved the overall application response time.