Impact Measurement and Intervention Strategies in Critical Systems Practice and Civic Tech

Impact Measurement and Intervention Strategies in Critical Systems Practice and Civic Tech

Introduction:

In both Critical Systems Practice (CSP) and the realm of Civic Tech, understanding impact and implementing effective intervention strategies is crucial. This article explores the importance of impact measurement in both fields and highlights actionable advice for measuring impact and constructing multimethodological intervention strategies.

Importance of Impact Measurement:

Measuring impact is vital for decision-makers, both within organizations and for external stakeholders. It serves three main purposes: informing internal decisions, driving improvements in tools, and improving communication of impact with partners.

1. Informing Internal Decisions:

By measuring the impact of tools, organizations can make informed decisions regarding resource allocation. Understanding the level of impact a tool has and how it compares to others helps prioritize where time and effort should be focused.

2. Driving Improvements in Tools:

Measuring the impact of civic tech tools allows organizations to identify how impact is being generated. This knowledge helps drive improvements in tools, ensuring they effectively serve their intended purpose. Traditional tech engagement metrics may not align with long-term impact, making it crucial to understand how tools are being used and by whom.

3. Improving Communication of Impact:

Effective communication of impact is essential for building strong partnerships. By measuring and understanding the impact of tools, organizations can effectively convey their value to partners and stakeholders. This helps establish credibility and foster collaboration.

Implementing Impact Measurement in Civic Tech:

Measuring impact in the Civic Tech space requires a focus on changes in end outcomes rather than solely relying on traditional tech engagement metrics. The following steps can be implemented to measure impact effectively:

1. Define and Validate User Groups:

Defining user groups based on the theory of change and how the tool is designed is essential. Implementing "exit intent" pop-ups on tools can help validate user groups by gathering data on who is using the tool.

2. Understand User Segments:

Combining aggregate data from tools with data from pop-ups provides insights into the number of users within each defined group. This information helps understand the reach and distribution of impact.

3. Evaluate Effectiveness for Each User Segment:

Incorporating questions within pop-ups that assess user satisfaction and goal achievement provides guidance on the effectiveness of tools for different user segments. This data helps identify areas of improvement and tailor interventions accordingly.

Insights from Impact Measurement in Civic Tech:

Through the implementation of impact measurement processes, valuable insights have been gained regarding both the impact of tools and the measurement process itself.

Impact Insights:

• The Wazimap tool caters to a wide range of users, including civil servants, private sector employees, citizens, academics, civil society employees, journalists, students, and elected officials.

• The tool is effective for approximately two-thirds of users but only one-third of journalists.

• The tool's effectiveness is significantly higher on desktop devices compared to mobile devices.

• Exploring the most useful features of the tool has further enhanced understanding of impact.

Process Insights:

• Responses to quick pop-ups have shown response rates ranging from 5-15%, depending on the tool and device.

• Periodic impact measurement through pop-ups, conducted quarterly, has proven to be sufficient, eliminating the need for continuous assessments.

• Embedding impact measurement processes and findings into teams' ways of working takes time and requires the development of a culture of measurement and learning.

Conclusion:

Measuring impact and constructing multimethodological intervention strategies are vital components of both Critical Systems Practice and Civic Tech. By understanding the importance of impact measurement, implementing effective measurement processes, and gaining insights from the measurement outcomes, organizations can drive positive change and effectively communicate their impact to stakeholders.

Actionable Advice:

• 1. Define and validate user groups for your tools to ensure a clear understanding of who the tool is designed for and how they will use it.

• 2. Evaluate the effectiveness of your tools for different user segments by incorporating user satisfaction and goal achievement assessments.

• 3. Implement periodic impact measurement through pop-ups to gather data on impact, eliminating the need for continuous assessments.

Remember, impact measurement is an ongoing process that requires dedication and a culture of measurement and learning to drive meaningful change.