EDM 2022 Author/Reviewer Checklist

EDM reviews this year will include questions related to fairness, equity, and positive social impacts. EDM 2021 featured a special session was focused on identifying concerns in the community, and resulted in a working group that drafted these guidelines, based on guidelines from several other conferences.

Authors should consider these guidelines during paper writing, since reviewers will be asked to provide feedback on these issues. Note that some guidelines may not be relevant to all papers, which is perfectly fine—e.g., a paper with no algorithms may have little to do with algorithmic bias.

1. Does the paper provide detail on the demographics of the population involved in the study? If so, does it indicate whether the demographic information is self-identified from participants or if identity characteristics are attributed to participants from some other source?
   • Providing the demographics of the population being studied, both if data are newly collected or if an existing dataset is being used, can provide insight into how representative the results may be for a general population.
   • Demographic characteristics that authors might want to consider describing include, but are not limited to: race/ethnicity, nationality, gender, age, grade, and socio-economic status. 
   • Self-identified characteristics are generally preferred over attributed characteristics, although if only attributed characteristics are available, providing these is more helpful than omitting any information.
   • If no information is available about the exact participants, some fallback options include providing demographic information at the school or district level.
   • To the extent that the research questions explicitly relate to demographic characteristics, authors should motivate why they focus on particular characteristics and the range of values included for these characteristics.
2. (For papers that collect new data from students or others): Does the paper describe how participants’ privacy rights were respected in the data collection process?
   • The policies specific to the context should be followed. For work conducted in the US or with US-based authors, this would typically include Institutional Review Board (IRB) approval if the work is considered research with human subjects. As another example, in the European Union, data should be handled in a way that is compliant with the General Data Protection Regulation (GDPR). For other countries, policies specific to the context should be followed.
3. Does the paper explicitly check for algorithmic bias in some fashion, and/or replicate new or existing results across population(s)? If so, does it motivate the identity characteristics examined and range of values used for these characteristics?
4. Does the paper include a discussion about potential positive and negative educational impacts of the proposed research artifact or application?
   • For negative impacts, it is likely helpful to consider possible harms in the following three scenarios: (i) when the technology is used as intended and functioning, (ii) when the technology is used as intended but giving incorrect results, and (iii) following from misuse of the technology.
   • It is particularly helpful to discuss whether any particular harms or benefits are likely to fall systematically on populations already experiencing marginalization or otherwise vulnerable.
5. Does the paper discuss ethical concerns related to the work, including potentially the framing of the task (e.g., ethical implications of (i) grouping together people into broad demographic/similar categories or (ii) assigning individuals or demographic/similar groups to experimental conditions)?
6. Does the paper discuss the limitations of the work?
   • Accurately describing the limitations helps to better situate the work, and it is expected that all research has limitations. Acknowledging these will be seen in a more favorable light than omitting or eliding them.
7. Does the paper describe the computational power needed for training or testing the given models? This could include time for training/testing, with information on the compute resources used, as well as time to run in actual deployment. Does the paper provide open-source code or datasets for the community?
   • Including information about the computational power needed can help inform both researchers and practitioners as to the feasibility of deploying the system in its current form in real educational settings, as well as provide valuable information about the reproducibility of results (i.e., if the needed computational resources are so large as to be available only at great time and expense, it may limit the replicability of the work by others). 
   • Ideally, code and datasets should be shared for better replicability.