The beginning of the COVID-19 pandemic brought significant change to how the humanitarian sector operates. As the world adapted to new health concerns, social distancing measures and global travel restrictions, monitoring and evaluation (M&E) practitioners faced a rapid shift from traditional in-person activities to remote approaches.
Now, remote digital M&E – often accompanied by greater reliance on local partners, in-country staff and third-party monitors – has become the ‘new normal’. But what are the ethical challenges that arise from this? And how can we address them?
Remote M&E is not new. In the past decade, it has enabled data gathering activities to continue in situations where evaluators and frontline staff would be at risk if working in person – for example, during Ebola outbreaks. The global nature of the COVID-19 crisis, however, has forced most organisations to change the way they work. For many, this has been an opportunity to further explore the potential benefits of remote methods and to garner a greater awareness of the gaps, challenges and ethical considerations that arise during their use.
Earlier this year I worked with ALNAP to organise a Skills-Building Workshop, during which humanitarian practitioners shared their experiences of conducting traditional M&E activities remotely during the pandemic. The accompanying research paper ‘Getting remote M&E right: ethics, challenges and gaps’ highlights the following five areas where there are ethical issues, challenges, gaps and limitations with remote M&E, before listing ways to improve in each. For a summary of the emerging good practice found in the paper, please expand each of the boxes below.
M&E practitioners can improve inclusion by focusing on the following:
Ensure that context, digital access and patterns of device use are properly understood. This is especially important when planning remote M&E to reach habitually excluded groups.
Actively design remote M&E plans for participation and inclusion. Include activities that are meaningful and valuable for the community.
Simplify tools, methods and approaches so that they are accessible and not too time consuming.
Complement remote digital M&E with socially-distanced, safe outreach by local staff, partners and community groups.
M&E practitioners can support safety and well-being by focusing on the following:
Avoid gathering unnecessary data. Asking affected communities to take part in M&E during highly stressful times such as a pandemic can put them under unnecessary stress and lead to survey fatigue.
Avoid or limit data-gathering on highly sensitive topics. We can never entirely control the privacy of a call or device and privacy is even more difficult to ensure during COVID-19 related restrictions.
Never assume confidentiality. Even when efforts are not addressing sensitive topics, respondents might not be alone while taking part in remote M&E surveys or online focus groups.
Do not keep women or children on the phone for extended periods, as this can trigger abusive situations. Always identify yourself and your reason for calling.
Be prepared to signpost people to available resources or help. People do not always stick to the topic of the survey – train local enumerators to be sensitive to other topics and equip them with a list of local resources.
Avoid transferring risk. Local staff and/or partner organisations should not be expected to conduct activities that were previously the responsibility of headquarter or country offices without added PPE, training, support, funding, capacity and compensation.
M&E practitioners can manage expectations and improve trust by focusing on the following:
Avoid extractive processes. Use M&E processes to stay in touch with communities, to engage with local actors and to build relationships of trust and support.
Build rapport. When reaching out to community members remotely, allow extra time to speak to people so that they feel comfortable before moving to data collection.
Focus data collection on needs that can be immediately resolved and feedback that can be addressed quickly. Needs stemming from the COVID-19 crisis are widespread. If an immediate response is impossible, clearly communicate what can be done or not.
Work with trusted local informants or experienced hotline staff who might be able to conduct socially-distanced and safe visits to gather community information and feedback.
Negotiate with donors to reduce the amount of data required for accountability. Push back against excessive data demands by consistently asking “why do we need this data? how will it be used?”
M&E practitioners can improve data quality by focusing on the following:
Adjust sampling, recruitment and other methods to reduce bias. Remote digital monitoring skews towards certain individuals.
Combine quantitative and qualitative data collection. For example, remote qualitative data can be collected by phone or WhatsApp to supplement quantitative data or check validity.
Weigh the potential for biased data with the value of having data right now. Teams conducting digital surveys will only be able to contact households that have a charged phone, meaning data collected will be biased towards those with more resources.
Assume a higher non-response rate when designing a sampling strategy. Use the level of non-response rates observed during piloting to inform adjustments to standard non-response rates.
Validate findings with communities. Organisations can be hesitant to contact local groups, but combining remote and in-person data collection with local guidance and direction is critical to understand the context.
M&E practitioners can improve data protection by focusing on the following:
Minimise the amount and sensitivity of data that is collected.
Obtain active and informed consent. Consider that the stress of COVID-19 might mean that truly informed and voluntary consent is not possible due to the circumstances.
Do a data risk–benefits assessment. If potential harms cannot be reduced to an acceptable level, do not continue with data collection and/or use.
Establish a data-sharing or data-processing agreement. This will ensure that responsibilities and liabilities for any data breach, unauthorised sharing, or other harm from data collection are clear.
Protect data throughout the full data life cycle. Ensure transparency about data use, retention, and eventual destruction.
While digital approaches have the potential to enable greater inclusion and outreach, we know they can exclude certain individuals and communities – generally the most vulnerable. What’s more, a lack of physical presence in the community can result in a lack of contextual awareness from M&E practitioners – leading to biased interpretations of the data and missed opportunities for inquiry.
Despite these concerns, some combination of in-person and remote M&E will likely be the norm throughout the pandemic and beyond. It is important for us to continue reflecting and improving our remote M&E practices so that if and when we are able to travel and meet in person again, we can make the most of both approaches based on the context in which we are working and the kinds of data we want to gather.