Balanced Scorecard approach to Software Quality
It essentially maps business strategy to customer,
finance, internal processes and learning & growth perspectives. More
information available at Balanced scorecard.
However, Balanced scorecard framework invoked a thought
process to apply the same to Software Quality. Below table applies Balanced scorecard to Software Quality.
Measurement or tracking of each Key Performance Indicators (KPIs) could be a
matter of debate as these could be tracked weekly, monthly, quarterly or
yearly.
Formulae listed in the below table can be tweaked
easily to individual needs. No guarantee of any sorts :). Few of these KPIs are available on internet on different software testing or quality assurance forums. This article tries to put these in balanced scorecard framework.
Assumption:
- We know how to calculate total cost of testing efforts
- There could be other KPI that individual leverage but I cannot list everything here. KPIs are for example purpose only.
Customer
|
|||||
Sr. No.
|
Objective
|
Measure
|
Target
|
Initiative
|
|
1.
|
Improve on features shipped
|
Number of Feature Request
|
Identify top 10 features requested by customers
|
1. Analyze Escalations, mailing lists, sales inputs
2. How many Escalations/mailing lists/sales inputs qualify as
features?
|
|
2.
|
Reduce critical bugs in production
|
Number of critical bugs reported by customer
|
Reduce critical bugs to 10% with respect to previous
release
Final goal: zero critical bugs in production
|
1. Analyze Escalations, customer reported incidents, mailing list etc.
|
|
3.
|
Improve product delivery Cycle time |
Automation productivity to expedite delivery
|
Improve the automation productivity consistently release
over release.
e.g.
Following targets could be considered:
1. Improve by 100%
2. Improve by 80%
3. Improve by 50%
|
1. Use following formula to track
automation productivity:
= (Total number of automated tests) / total automation
efforts
|
|
4.
|
Improve product delivery Cycle time |
Test Cycle time
|
Reduction in total testing time
|
1. Use following formula to track
reduction in total test cycle time
= Total testing downtime/ Total test execution time
|
|
5.
|
Innovation
|
Number of new ideas generated
|
Improve on idea implementation into product
|
1. Use following formula to track
reduction in total test cycle time
= Number of new ideas (suggestions) /
Ideas (Suggestions) implemented
|
|
Finance
|
|||||
Sr. No.
|
Objective
|
Measure
|
Target
|
Initiative
|
|
1.
|
Reduce Software testing cost
|
Identify cost per test case
|
Reduce cost per test case
|
1. Use following formula to track
cost per test case
= Total cost / number of test cases
|
|
2
|
Reduce Software testing cost
|
Identify cost per automated test case
|
Reduce cost per automated test case
|
1. Use following formula to track
cost per test case
= (Total automation cost) / (number of automated test
cases)
|
|
3
|
Reduce cost of minor releases
|
Identify release cost
|
Reduce cost of minor release as compared to previous minor
releases
|
1. Use following formula to track
cost per release
= Minor (No. Of release defects filled + No. Of release
resources) / Major (No. Of release defects filed + No. Of release resources)
+ Minor (No. Of release defects filled + No. Of release resources)
2. Consistently track the output
over all releases
|
|
4
|
Reduce cost of major releases
|
Identify release cost
|
Reduce cost of minor release as compared to previous minor
releases
|
1. Use following formula to track
cost per release
= Major (No. Of release defects filled + No. Of release
resources) / Major (No. Of release defects filed + No. Of release resources)
+ Minor (No. Of release defects filled + No. Of release resources)
2. Consistently track the output
over all releases
|
|
Internal processes
|
|||||
Sr. No.
|
Objective
|
Measure
|
Target
|
Initiative
|
|
1.
|
Improve test effectiveness
|
Measure Automation percentage
|
Improve overall automation percentage release over release
to bring more test effectiveness
|
1. Use following formula to
improve test effectiveness by measuring percentage of automation
= (Automated tests) / (Manual tests + Automated tests)
|
|
2.
|
Improve test effectiveness
|
Measure manual percentage
|
Reduce manual efforts close to zero
|
1. Use following formula to
improve test effectiveness by measuring percentage of manual tests
= (Manual tests) / (Manual tests + Automated tests)
|
|
3.
|
Improve test effectiveness
|
Defect slippage to internal customers and in production
(consider deferred incidents as well)
|
Zero Defect slippage to internal customers and in
production (consider deferred incidents as well)
|
1. Use following formula to
improve test effectiveness by measuring defect slippage
= (No. Of defects by internal customers) / (No. Of
defects by internal customers+ No. Of deferred defects)
|
|
4.
|
Improve software testing quality
|
Defect Severity distribution
|
Zero high severity incidents
|
1. Use following formula to
improve software testing quality
= (No. Of Sev 1 and Sev 2 defects) / (Total number of
defects)
|
|
5.
|
Improve test effectiveness (Operations)
|
Measure number of parallel qualifications as compared to
major/minor releases
|
Ideal targets could be e.g.
1. Less than Number of
major and minor releases combined
2. Less than m
|
1. Use following formula to
improve test effectiveness by measuring number of qualifications in progress
= (Number of non-release qualifications) / (Number of
major releases + Number of minor releases)
|
|
6.
|
Improve documentation
|
Measure level of completeness, accuracy, simplicity
|
Improve quality of documentation
|
1. Number of new documents per
release
2. Number of new updates per
document per release
3. Percentages of new updates with
respect to new features
|
|
7.
|
Improve software testing quality
|
Tests passing per build (daily)
|
Set following targets for percent of tests passing per
daily build e.g
1. Less than 10 %
2. Less than 5 %
3. Less than 2%
|
1. Use following formula to
improve test effectiveness by measuring number of qualifications in progress
= (Number of tests passed) / (Total number of tests
planned for execution)
|
|
8.
|
Improve software testing quality
|
Features per iteration (or release)
Measure story points per iterations in Agile model
|
Measure how many features that we ship as a part of each
iteration (or release)
|
1. Use following formula to
measure features in iterations
= (Number of features planned in iteration or release) / (Total
number of targeted features)
|
|
9.
|
Improve on business value deliverables
|
How early we can deliver a value in the release?
|
In Agile Model, we have to deliver high value in early
sprints.
|
1. Use following formula to
measure features in iterations
Customer value of feature x= Cx points
Storypoint of a feature x = Sx points
% Business value of feature x = Cx / Sx
|
|
Learning and Growth
|
|||||
Sr. No.
|
Objective
|
Measure
|
Target
|
Initiative
|
|
1.
|
Resource availability
|
If accepted for qualification then how much number of
qualified resources should be available?
|
We should have sufficient number of resources available to
work on every qualification.
|
1. Use following formula to
identify trained resources that will work on release qualification
= (Resources to work on qualification) / (total number of
available resources)
|
|
2.
|
Knowledge transfer/ Transfer of Information
|
How many knowledge transfers (Transfer of Information) are
happening within testing organization.
|
At least 1 KT (ToI) per week within team
|
1. Use following formula to
identify progress on knowledge transfer in our teams
= (ToI delivered) / (Total number of ToI delivered + Total
number of ToI available in pipeline)
|
|
3.
|
Cross-team collaboration
|
Measure how much efforts were spent by your team (A) to
collaborate with other teams (B)
|
Team collaboration should be 100%
|
1. Use following formula to
measure team collaboration,
= (Total efforts spent by your team: A) / (total efforts
spent by our team: A + total efforts spent by other teams:B)
|
|
As I said above table lists KPIs that can be tweaked to individual needs and but I cannot guarantee their implementation lead to success.
If you want to suggest correction (I think there is scope of improvement) or suggest additional KPIs in each perspective above then leave a comment below.
Never miss an update. Subscribe and follow to stay informed. Delivered Every Tuesday.
We hate spam too, we will never share your details.
.jpg)
.jpg)
.jpg)
Hello,
ReplyDeleteThe Article on balanced Scorecard Approach to Software Quality is amazing give detail information about it.Thanks for Sharing the information about report of Software Testing.
Software Testing Company
Thanks for sharing your work on balanced scorecard. It will be very useful for me to start from your example
ReplyDeleteThanks for sharing your work. It will be very helpful for me to apply it to my case.
ReplyDelete