In Part 1 of the “Delivery as a Product” post, I described how important it is to capture and analyze metrics about your delivery pipeline. This is critical because one of the main goals of this productization of delivery is to relentlessly look for improvements. And you know how the old adage goes – “you can’t improve what you can’t measure”.
In Part 2 I go deeper into those critical metrics and KPIs I often set as core principles and pillars for my software and product teams. A few weeks ago I was listening to The State of DevOps podcast and I was pleasantly surprised that the way they had categorized their industry findings and key metrics closely resembled what our teams were already doing, or at least focusing on. Talk about confirmation bias.
Throughput
In general, it can be visualized as how quickly you can develop and deploy features to Production, focusing on bringing value to the user as quickly as possible. In reality, throughput is an umbrella KPI that consists of several other metrics:
Lead time
Lead time is the measurement of how much time passes between task creation and when the work is completed. A common mistake is to think about it as just a development metric, but you should consider it from the user’s point of view. How long it takes, from the perspective of the user, for a feature to go from “requested” to “complete.”
Deployment frequency
The cadence at which you deploy and deliver value to the user. This could be daily, weekly, bi-weekly, monthly, etc.
Deployment Speed
During deployment, there are several steps that need to happen, and typically there are several environments involved as well. Deployment speed is the metric that captures the time it takes to perform a deployment process from start to finish, going over all the necessary steps (e.g., moving code and content between environments, system integration testing, UAT, production cleanup, database backups, etc.).
Release automation
A quantitative and qualitative assessment of your release management practices. Ideally, you would like to automate everything in the process that is repeatable. The benefits of an automated release management practice include higher release velocity, reduction of errors, consistent releases, a better experience for team members, predictability, etc.
Quality: Testing, Stability, and Reliability
Quality is key in assessing your delivery practices. You can argue that it’s easy to improve and have a great throughput if you are ignoring the testing and validation requirements. Automated testing for functionality, content, and compliance is usually harder to nail down to the point you can sleep at night when unsupervised deployments are happening on a nightly basis.
A couple of considerations related to testing:
- The closer to Production you find an issue/bug, the costlier it is. Therefore, make all the necessary adjustments and efforts to try to find issues as early as possible.
- This might mean updating your team’s definition of done (DoD) – including unit testing, acceptance criteria, peer reviews, etc., to your development process. It’s a developer’s responsibility to make sure the work completed conforms to the definition of done. It’s not QA’s responsibility to find issues related to incomplete work, misunderstood scope, etc.
Key quality-related KPIs
Test speed
How much it takes to complete each phase of testing? (e.g., smoke testing by environment, staging validation, system integrating testing, production validation, etc.)
Test coverage
How much of the entire application, product, or website is covered by your test plan, test cases, unit testing, etc?
Test automation
How much of your application is covered by automated testing? How far are you from true unsupervised automated deployments?
Incidents caused by a release
Releases cause changes to a system, inevitably some changes will have an impact on the production environment.
Number of needed rollbacks
Sometimes you can repair any production issues without the need to roll back to a previous stable version, unfortunately, there are other cases this is not possible without an extended downtime period.
Release downtime & mean-time-to-repair
Plan for unexpected downtime and add mitigations to your process.
Operational Excellence
Operational excellence is a mindset that embraces certain principles and tools to create a culture of excellence within an organization. Operational excellence means every team member can see, deliver, and improve the flow of value to a customer. In the world of product and software development, principles to achieve operational excellence can be divided into 3 components: people, processes, and technology/tools.
This is a very broad topic. Books have been written about this and the areas to focus on are very unique to your own organization and teams. My suggestion is to do an honest and quick assessment of your general delivery practice and try to identify what areas are the ones causing most of the headaches. You can use the following list of common reasons why projects fail and try to apply it to your own situation and environment.
| # | Reason Why Projects Fail |
|---|---|
| 1 | Unclear business requirements |
| 2 | The solution design and the final application did not match the business requirements |
| 3 | Substandard planning and execution |
| 4 | Lack of standardized production processes |
| 5 | Working in silos |
| 6 | Lack of agile practices |
| 7 | Friction caused by undefined roles |
| 8 | Lack of discipline |
| 9 | Inadequate testing and quality assurance |
| 10 | Focusing on technical detail instead of business value |
Key operational excellence-related KPIs
For operational excellence, it’s harder to get to a set of quantitative metrics, but a qualitative assessment of the maturity of your practice can take you a long way.
| Repeatable | Requirements management |
| Software Project Planning | |
| Software Project Tracking and Oversight | |
| Software Quality Assurance | |
| Software Configuration Management | |
| Defined | Organization Process Focus |
| Organization Process Definition | |
| Training Program | |
| Integrated Software Management | |
| Software Product Engineering | |
| Intergroup Coordination | |
| Peer Reviews | |
| Managed | Quantitative Process Management |
| Software Quality Management | |
| Optimizing | Defect Prevention |
| Technology Change Management | |
| Process Change Management |
Conclusion
It’s important to find a way to capture metrics and data about your delivery practice. Remember, you can’t improve what you can’t measure. Capture, analyze and compare throughout the lifespan of your product/project, and do it frequently and with a sensible cadence.
