We’ve been reading all your input regarding what goals we should set for a successful forum, and we’d like to get your feedback now on the evaluation plan for this forum. An evaluation plan broadly includes a theory of change-- how we think this forum will work to impact the Wikimedia ecosystem-- and a plan to test that theory.
Here is a picture of our theory of change for this forum (image text is in English). I’m including the text below as well for translation and some context for how we think about the theory of change.
Output (This is the Wikimedia Foundation’s direct “deliverable” to our communities): Deployment, support, communication about, and administration of a platform for multilingual Movement Strategy collaboration and communication.
Cognitive Outcomes (These are changes in beliefs or knowledge that, in theory, would arise from the “Output”):
- Potential Movement Strategy contributors will be more aware of the Movement Strategy process, especially newcomers, those speaking languages other than English, and those living in geographic regions underrepresented in our movement.
- Potential and existing Movement Strategy contributors will be aware of the collaboration platform.
- Existing and potential Movement Strategy contributors will feel comfortable and confident using the MS Forum, especially newcomers, those speaking languages other than English, and those living in geographic regions underrepresented in our movement.
Action Outcomes (This is a change in actions or behaviors that, in theory, would arise from cognitive outcomes): More people will contribute to Movement Strategy implementation through the MS Forum, particularly newcomers, those in underrepresented geographies, and those living in geographic regions underrepresented in our movement.
Impact (This is a change in the conditions of our movement that, in theory, would arise from the short and mid-term outcomes): Movement Strategy implementation will be more collaborative and inclusive of newcomers and those in underrepresented geographies.
Now that we have a drafted theory of change, we can decide how to measure whether our theory is correct.
Measuring Outputs: MS Forum is live and supported.
- Wikimedia Foundation staff ensure the platform remains active and moderated, communicates about it to potential collaborators, and includes features that enable participation across experience level, geography, and language diversities.
Measuring Cognitive Outcomes:
1. Awareness of Movement Strategy
- Change in awareness of Movement Strategy process among newcomers, Wikimedians not fluent in English, and Wikimedians living in geographic regions underrepresented in our movement (source: Community Insights survey Oct 2020, Jun 2021).
2. Awareness of MS Forum
- Number of new user accounts created each month, particularly among Wikimedians living in geographic regions underrepresented in our movement (source: Forum admin data)
- MS Forum traffic each month, particularly among viewers living in geographic regions underrepresented in our movement (source: Forum admin data).
3. Comfort with Discourse Platform
- Percentage of MS Forum users satisfied with ease of use, accessibility, safety, and platform utility for Movement Strategy collaboration particularly among Wikimedians living in geographic regions underrepresented in our movement, those not fluent in English, and newcomers (source: brief survey of MS Forum users in January and July of 2023).
Measuring Action Outcomes: More people will contribute to Movement Strategy collaboration on the MS Forum.
- Monthly number of active users, particularly those living in geographic regions underrepresented in our movement (source: Forum admin data) in comparison to meta.
- Percentage of users not fluent in English, percent of users who are newcomers, Wikimedia-related roles of users (eg., movement organizer, affiliate leader or representative, content contributor, etc) (source: brief survey of forum users in January and July of 2023).
- Diversity of active discussion topics (source: qualitative coding of forum topics, survey of forum users in January and July of 2023).
Measuring Impact: Movement Strategy implementation will be more collaborative and inclusive of newcomers, those not fluent in English, and those living in geographic regions underrepresented in our movement.
- Number of Movement Strategy-related problems solved on MS Forum, including controversial ones (source: Forum tags and survey of forum users).
- Number of new groups/coalitions/collaborations emerging from MS Forum (source: Forum tags and survey of forum users in January and July of 2023).
- Number of new grant proposals (and funded proposals) organized on MS Forum, particularly among those living in geographic regions underrepresented in our movement (source: Forum tags and survey of forum users).
We invite your feedback on both the theory of change and the evaluation plan!