02 Jul Looking in the mirror: Effective evaluation of social labs

Three students from the USA, Germany and Vietnam, recently completed a Masters Thesis which looks at ways that social labs like the Southern Africa Food Lab are and could be evaluated. It was written as part of the Master´s program “Strategic Leadership towards Sustainability”, a 10-month program in Karlskrona, Sweden that focuses on advancing students’ skills and knowledge to build their capacity to be strategic leaders for sustainable societies. The thesis, titled Looking in the Mirror – Social Labs and Evaluating in Complexity by Robin Woolner, Robin Dirks and Trang Nguyen on one hand compared how different social labs are already using evaluation and on the other hand applied an “adaptive capacity” lens as a way to indicate the resilience of a social lab. They summarize their recommendations what to consider for an evaluation strategy and for evaluating adaptive capacity:

1. Evaluation strategy & purpose of evaluation

An evaluation strategy depends on the challenge that the lab is trying to address, as well as on the Theory of Change of the convening organization. To develop a strategy, firstly, the purpose and evaluation approach needs to be defined: What are the most important evaluation outcomes? Is the focus to support learning and to develop the lab practice (developmental evaluation) or to build legitimacy and prove impact (summative evaluation)? What does success of the evaluation mean? What are expectations or desired outcomes of different lab stakeholders?

Some of the different roles or “dimensions” that an evaluation can fulfill are:

• Process and workshop development: How well did the workshop design accomplish its desired outcomes? What worked well, what didn´t? What could be improved? Tools include feedback forms, post-workshop interviews with participants and after-action review calls with the facilitation team to reflect how “the learning landed”.
• Insights about the system: what is the lab team learning about the system? How does that inform the lab challenge? This is closely linked to the harvesting of the lab process, using tools like graphic harvesting, learning journeys and dialogue interviews.
• Impact on participants: how have behaviors and relationships changed? Tools include network analysis, continuous interviews with a set of the same participants
• Effectiveness of lab outcomes: how innovative are the prototypes really? What are tangible success stories? Since the outcomes of a social lab process cannot be predetermined,
• The long-term impact on the larger system: how does the work of the lab influence wider systems change? Hereby, broader development indicators relevant for the lab challenge (e.g. household income, number of unemployed people, malnourished children etc.) could be correlated with the impact of the social lab.
• Harvesting and communication: how will the results be harvested and how will they be communicated? To communicate the outcomes, an online archive could make all the content public and photos, videos and material could be published. Furthermore, (digital) storytelling or blogs can be a way to keep the public informed.

2. Evaluation of Adaptive Capacity

The students looked at ways to evaluate the adaptive capacity of a social lab, which indicates its resilience to deal with complex challenges. They synthesize three key aspects that could be continuously evaluated and serve as a “mirror” to indicate the progress of a social lab in terms of resilience:

1. Trust amongst lab members: Trust seems to be a key feature for the success of a social lab, as it requires highly diverse perspectives and world views to co-exist; trust seems to be an important pre-requisite for a group to embrace diversity and leverage tensions that may arise. It can be evaluated both through observation (e.g. how do participants deal with tension?) as well as by conducting interviews.
2. Prototyping Capacity: Prototyping capacity refers to the ability of lab members to continually test and refine their assumptions as well as their solutions based on trial and error. Hereby, reflective practices play an important role for a lab to be responsive and flexible. This aspect can also be evaluated through observation (do people start small and test their assumptions?) as well as through interviews asking about course-corrections for example.
3. Systems Perspective: A systems perspective is important for a social lab to address the root causes of the lab challenge, from where innovative solutions can be developed. This could be observed for example by looking at how participants are framing the challenge and issues around the challenge (are they paying attention to the relationships, perspectives and dynamics?) as well as whether they incorporate multiple perspectives in the solutions they are finding. Furthermore, continuous interviews with participants could look how the individual insights and perspectives change over a period of time.

These aspects could serve as “success principles” beyond predetermined impact measurements to enable a “principle-focused” evaluation, adjusted to the specific context of each social lab. Unlike traditional summative evaluation, a principle-focused evaluation is capable of evaluating progress in a complex situation, where measurable “numbers” can be misleading. Herby, it can be evaluated in how far the principles are being adhered to. The above “adaptive capacity aspects” could help social labs as a starting point to formulate their own principles and to evaluate these.

Access the full thesis here: Looking in the Mirror – Social Labs and Evaluating in Complexity