Balanced Scorecard approach to Software Quality


Balanced Scorecard approach to Software Quality

Few months back, I was introduced to Balanced scorecard concept by Rajul @Sunstone.
It essentially maps business strategy to customer, finance, internal processes and learning & growth perspectives. More information available at Balanced scorecard.

However, Balanced scorecard framework invoked a thought process to apply the same to Software Quality.  Below table applies Balanced scorecard to Software Quality. Measurement or tracking of each Key Performance Indicators (KPIs) could be a matter of debate as these could be tracked weekly, monthly, quarterly or yearly.

Formulae listed in the below table can be tweaked easily to individual needs. No guarantee of any sorts :).  Few of these KPIs are available on internet on different software testing or quality assurance forums. This article tries to put these in balanced scorecard framework.

Assumption:
  • We know how to calculate total cost of testing efforts  
  • There could be other KPI that individual leverage but I cannot list everything here. KPIs are for example purpose only.
 
Customer
Sr. No.
Objective
Measure
Target
Initiative
1.
Improve on features shipped
Number of Feature Request
Identify top 10 features requested by customers
1.     Analyze Escalations, mailing lists, sales inputs
2.     How many Escalations/mailing lists/sales inputs qualify as features?
2.
Reduce critical bugs in production
Number of critical bugs reported by customer
Reduce critical bugs to 10% with respect to previous release

Final goal: zero critical bugs in production
1.     Analyze Escalations, customer reported incidents, mailing list etc.
3.
Improve product delivery Cycle time
Automation productivity to expedite delivery
Improve the automation productivity consistently release over release.
e.g.
Following targets could be considered:
1. Improve by 100%
2. Improve by 80%
3. Improve by 50%
1.     Use following formula to track automation productivity:

= (Total number of automated tests) / total automation efforts
4.      
Improve product delivery Cycle time
Test Cycle time
Reduction in total testing time
1.     Use following formula to track reduction in total test cycle time

= Total testing downtime/ Total test execution time
5.      
Innovation
Number of new ideas generated
Improve on idea implementation into product
1.     Use following formula to track reduction in total test cycle time

= Number of new ideas (suggestions) /
Ideas (Suggestions) implemented





 Finance
Sr. No.
Objective
Measure
Target
Initiative
1.      
Reduce Software testing cost
Identify cost per test case
Reduce cost per test case
1.     Use following formula to track cost per test case

= Total cost / number of test cases
2
Reduce Software testing cost
Identify cost per automated test case
Reduce cost per automated test case
1.     Use following formula to track cost per test case

= (Total automation cost) / (number of automated test cases)
3
Reduce cost of minor releases
Identify release cost
Reduce cost of minor release as compared to previous minor releases
1.     Use following formula to track cost per release

= Minor (No. Of release defects filled + No. Of release resources) / Major (No. Of release defects filed + No. Of release resources) + Minor (No. Of release defects filled + No. Of release resources)

2.     Consistently track the output over all releases
4
Reduce cost of major releases
Identify release cost
Reduce cost of minor release as compared to previous minor releases
1.     Use following formula to track cost per release

= Major (No. Of release defects filled + No. Of release resources) / Major (No. Of release defects filed + No. Of release resources) + Minor (No. Of release defects filled + No. Of release resources)

2.     Consistently track the output over all releases





Internal processes
Sr. No.
Objective
Measure
Target
Initiative
1.      
Improve test effectiveness
Measure Automation percentage
Improve overall automation percentage release over release to bring more test effectiveness
1.     Use following formula to improve test effectiveness by measuring percentage of automation

= (Automated tests) / (Manual tests + Automated tests)
2.
Improve test effectiveness
Measure manual percentage
Reduce manual efforts close to zero
1.     Use following formula to improve test effectiveness by measuring percentage of manual tests

= (Manual tests) / (Manual tests + Automated tests)
3.
Improve test effectiveness
Defect slippage to internal customers and in production (consider deferred incidents as well)
Zero Defect slippage to internal customers and in production (consider deferred incidents as well)
1.     Use following formula to improve test effectiveness by measuring defect slippage

= (No. Of defects by internal customers) /  (No. Of defects by internal customers+ No. Of deferred defects)
4.
Improve software testing quality
Defect Severity distribution
Zero high severity incidents
1.     Use following formula to improve software testing quality

= (No. Of Sev 1 and Sev 2 defects) / (Total number of defects)
5.
Improve test effectiveness (Operations)
Measure number of parallel qualifications as compared to major/minor releases
Ideal targets could be e.g.
1. Less than Number of major and minor releases combined
2. Less than m
1.     Use following formula to improve test effectiveness by measuring number of qualifications in progress

= (Number of non-release qualifications) / (Number of major releases + Number of minor releases)
6.
Improve documentation
Measure level of completeness, accuracy, simplicity
Improve quality of documentation
1.     Number of new documents per release
2.     Number of new updates per document per release
3.     Percentages of new updates with respect to new features
7.      
Improve software testing quality
Tests passing per build (daily)
Set following targets for percent of tests passing per daily build e.g
1. Less than 10 %
2. Less than 5 %
3. Less than 2%
1.     Use following formula to improve test effectiveness by measuring number of qualifications in progress

= (Number of tests passed) / (Total number of tests planned for execution)
8.      
Improve software testing quality
Features per iteration (or release)

Measure story points per iterations in Agile model
Measure how many features that we ship as a part of each iteration (or release)
1.     Use following formula to measure features in iterations

= (Number of features planned in iteration or release) / (Total number of targeted features)
9.      
Improve on business value deliverables
How early we can deliver a value in the release?
In Agile Model, we have to deliver high value in early sprints.

1.     Use following formula to measure features in iterations

Customer value of feature x= Cx points
Storypoint of a feature x = Sx points
% Business value of feature x = Cx / Sx





Learning and Growth
Sr. No.
Objective
Measure
Target
Initiative
1.      
Resource availability
If accepted for qualification then how much number of qualified resources should be available?
We should have sufficient number of resources available to work on every qualification.
1.     Use following formula to identify trained resources that will work on release qualification

= (Resources to work on qualification) / (total number of available resources)
2.      
Knowledge transfer/ Transfer of Information
How many knowledge transfers (Transfer of Information) are happening within testing organization.
At least 1 KT (ToI) per week within team
1.     Use following formula to identify progress on knowledge transfer in our teams

= (ToI delivered) / (Total number of ToI delivered + Total number of ToI available in pipeline)
3.      
Cross-team collaboration
Measure how much efforts were spent by your team (A) to collaborate with other teams (B)
Team collaboration should be 100%
1.     Use following formula to measure team collaboration,

= (Total efforts spent by your team: A) / (total efforts spent by our team: A + total efforts spent by other teams:B)







As I said above table lists KPIs that can be tweaked to individual needs and but I cannot guarantee their implementation lead to success. 

If you want to suggest correction (I think there is scope of improvement) or suggest additional KPIs in each perspective above then leave a comment below.

Note: We at TechSutram take our ethics very seriously. More information about it can be found here.
Mandar Pise Opinions expressed by techsutram contributors are their own. More details

Mandar is a seasoned software professional for more than a decade. He is Cloud, AI, IoT, Blockchain and Fintech enthusiast. He writes to benefit others from his experiences. His overall goal is to help people learn about the Cloud, AI, IoT, Blockchain and Fintech and the effects they will have economically and socially in the future.

3 comments:

  1. Hello,
    The Article on balanced Scorecard Approach to Software Quality is amazing give detail information about it.Thanks for Sharing the information about report of Software Testing.
    Software Testing Company

    ReplyDelete
  2. Thanks for sharing your work on balanced scorecard. It will be very useful for me to start from your example

    ReplyDelete
  3. Thanks for sharing your work. It will be very helpful for me to apply it to my case.

    ReplyDelete

    Your valuable comments are welcome. (Moderated)