Pre-registrations for remote research through Discoveries Online

Our research team pre-registers all new projects through the Open Science Framework (for recent examples, see here and here). Below, we’ve provided a template for the language we use in pre-registrations to describe our various protocols for online data collection.

Sample Pre-Registration Text

Data Collection Procedures

Participants will complete the study on their home or personal computers via the Discoveries Online research platform, developed specifically for remote developmental research (Rhodes et al., 2020). This platform captures both video and survey data from participating families. Participants on the platform are recruited in a multitude of ways, including but not limited to: community outreach events, social media advertising, posts on Prolific, and volunteer sign-ups at in-person data collection sites. For this particular study, we will only recruit participants with children between the ages of 4.5-7.99 years. Parents of eligible children will be contacted via email and offered the opportunity to log-on and participate. Children and parents will then follow an IRB-approved consent and assent procedure. The study will be presented as a series of short videos and verbally narrated questions, which children will respond to by clicking on the designated “buttons” to indicate their response. If children are unable to maneuver the mouse and click on the buttons themselves, parents will be encouraged to do the clicking for children (without altering their children’s responses or providing feedback in any way). At the end of the session, families will be compensated with a $10 Amazon gift card for their participation.

Stopping Rule

We plan to stop data collection in the week when we have obtained usable data from at least 220 participants in each sample (or once we have obtained usable data from 50 participants per condition; whichever comes first). We will intentionally over-sample in this way to account for drops, which occur at a rate of approximately 10% of the full sample for online data collection based on previous work (Leshin, Leslie, & Rhodes, under review).

Data Exclusion

All videos of participating children will be coded to make sure a visible child is participating in order to be included in the sample and that a parent provides verbal consent. Previous unmoderated remote research with children using similar methods has found very low rates of parent or sibling interference (less than 1% of trials), and excluding such trials had no consequence for the overall pattern of findings ( Leshin et al., under review). Therefore, we will randomly select 20% of videos to be coded by a trained researcher in order to estimate the interference rate. If it is within the bounds of the interference level found in previous work, we will conclude that the interference level is negligible and highly unlikely to influence the study patterns and therefore retain all trials for analyses. However, if the interference rate in the subset of coded videos is more than the rate estimated from previous work, then we instead code every video trial-by-trial for instances of parental interference (with 20% also checked by a reliability coder), and we will exclude any trials where the parent or a sibling interferes such that there is a question of whether the answer reflects the child’s response. If more than 25% of a child’s responses are excluded for this reason, the participant’s data will be dropped from analyses. Our guide to interference coding is available here (note: this guide is currently under revision, and an updated version will be posted to this site as soon as it is ready).

We do not plan to exclude based on children's success on manipulation checks and comprehension questions; however, if more than 25% of our sample fails these, we will conduct planned exploratory analyses including just the subsample of participants who passed the respective measures. We plan to report the percentage of participants who passed and did not pass these comprehension checks.

Missing Data

Partial data from participants will be included if at least 50% of items are complete; participants with less than 50% completion will be excluded. We also plan to exclude any participants who are not within our age-range of 4.5-7.99 years old. For partial data that is included, we plan to just look at scores out of total responses completed.