A critique of metrics
Metrics are measurements or indicators used to evaluate the performance or effectiveness of something, such as a business, a program, or a system. There are several type of metrics that you might encounter
- Financial metrics: These metrics are used to measure the financial impact, eg income, expense, profits, GMV, Unit metrics etc
- Operational metrics: These metrics are used to measure the operational performance of a business, such as CSAT, KISS Metrics, NPS, earning per employee etc.
- Performance metrics: These metrics are used to measure the performance of an individual, a team, or an organization, such as time to launch, time from idea to landing, lines of code etc. For software systems it might also be time it takes for a user to complete a task in the system, system load for the transaction, number of concurrent transactions per second, users severed per second etc.
- Quality metrics: These metrics are used to measure the quality of a product, service, or process, such as bugs per kLOC, customer satisfaction, and compliance with standards.
- Metrics can be misleading: The use of metrics can be subject to various forms of bias, such as selection bias, measurement bias, or interpretation bias. This can lead to inaccurate or incomplete conclusions about the performance or effectiveness of something.
- Metrics can be oversimplified: The use of metrics can often lead to oversimplification of complex issues or situations. This can result in incomplete or distorted understandings of the underlying phenomena being measured.
- Metrics can be used for the wrong reasons: The only things that get measured get improved. The use of metrics can be motivated by the desire to achieve a specific outcome or result, rather than by a genuine desire to understand the underlying phenomena. This can lead to the misuse or abuse of metrics for ulterior purposes.
- Metrics can be subject to gaming: Anything that can be gamed, will be gamed. The use of metrics can incentivize people or organizations to engage in behaviors that improve their metrics, even if these behaviors are not beneficial in the long term. This can lead to short-term gains at the expense of long-term success.
So I would argue that while metrics can be a useful tool for evaluating performance or effectiveness, they should be used with caution and consideration of their limitations and potential biases.