Hosting studies on Discoveries Online

As part of creating a system for unmoderated remote research for our lab, we developed an online platform for presenting studies to families via their internet browsers, recording the webcam data, and uploading the videos securely. In our own lab, we have integrated this platform for data collection with a larger system for participant recruitment and management. While the participant database and management systems are unique to our lab, the component of the platform that presents the study and securely stores video data, Discoveries Online, can host studies for other labs as well, and may present a good option for labs who are first trying out this approach and do not want to create their own video-recording and storage platform.

Discoveries Online will give you a URL that you can distribute to families directly via email, post on childrenhelpingscience.com, Prolific, or another platform, or link to from your own recruitment website. We highly recommend creating a landing page in your own website with information about your research before directing participants to the study link. When families click on the link that we give you, it will take them to a page that requests permission to activate their webcam, and once activated, will begin recording. Please advise your participants to use ONLY either Google Chrome or Mozilla Firefox as their default web browser for Discoveries Online studies. The first part of your experiment should contain your consent procedures (for example, here's how we obtain consent). At the end of the study, you should include these instruction slides for parents to upload their videos.

We will provide you with a research account and log-in, where you will be able to access your videos, which will be temporarily stored securely (and accessible only by you) on an external web server (more info on this coming soon). You will need to promptly download them and store them locally as required by your IRB, at which point, we ask you to manually erase them from the server using the "delete" button within 3 days of downloading them. Please note that some sessions may suffer from upload issues for each study due to a combination of variables like study video size, participant computer and network configuration, and web browser.

We detail below how to get your study running on Discoveries Online and how to access your study videos once a family has finished participating.

Preparing to host your study on Discoveries Online

We are currently piloting hosting studies for other labs. If you are interested in having a study hosted in the near future, please email our lab manager, Daryl, just to let us know that you are interested, and he can give you a time-frame for when we might be able to accommodate your study.

We currently can host studies that (1) have IRB approval for remote data collection that is consistent with our methods, (2) include consent and child assent at the start of the programmed study, and video upload instructions at the end, (3) are programmed in Qualtrics or a similar platform and can receive an embedded ID, (4) can play in a Firefox or Chrome browser, and (5) are no more than 20 minutes long.

When you are ready for us to host your study, please email these things to our lab manager, Daryl:

  • Confirmation of your IRB approval for remote data collection (please attach all relevant documents)
  • Confirmation that you have included informed consent and child assent at the beginning and video upload instructions at the end
  • Confirmation that your study is programmed to accept an embedded ID
  • Confirmation that you understand that studies will only work in a Chrome or Firefox browser and have a plan for how to communicate that with your participants
  • Approximately how long your study should take families to complete
  • A description of your plan for recruitment and whether you plan to provide participant incentives
  • The link(s) for your programmed experiment

Daryl will confirm that he received your materials and when we’ll be able to put up your study. Once it is up, we will email you:

  • A link to the study that you can distribute to families
  • Log-in credentials that you will be able to use to download your videos
Each lab must handle their own recruitment, participant management and communication, and incentives. We highly recommend that you create a landing page on your own website to provide information about your research to parents before having them click the study link. You might choose to recruit for your study on childrenhelpingscience.com or Prolific.

Linking Qualtrics surveys to Discoveries Online

If you are setting your study in Qualtrics, you should embed participant ID into your survey flow so that you can match participants’ IDs in Discoveries Online (i.e., their video data) to their ID in your exported Qualtrics data (i.e., their response data). To do so, simply add an Embedded Data block in your study’s survey flow and select Add a New Field. Name the new field as “ID” and leave it so that the Value will be set from Panel or URL. Then, drag the block to the beginning of the survey flow. This is what the block should look like.

That way, when a participant responds through Discoveries Online, a unique ID will be generated and will be embedded into both their video data and their response data (as shown below).


Posting on Prolific

Prolific is an online data collection platform where anyone can post (or participate in) studies. Below, we outline how you can post an unmoderated developmental study (like those hosted through Discoveries Online) on Prolific.

Before posting on Prolific, there are a couple of Prolific-specific adjustments you will first have to make to your Qualtrics survey:

  1. You will likely need to ensure that you have child-specific demographic questions included. Note, however, that Prolific is very specific about the kinds of demographic information that can be collected, so you cannot ask for birthdate (only birth year and month). In our lab, we usually include these child demographic items: child’s birth month and year, child’s gender, child’s race/ethnicity, and child’s first language.
  2. You will have to ensure that you have the embedded ID set up properly per usual (see above section for instructions).
  3. (*See also Step #7*) You will need to set up your study in Qualtrics so that it redirects to a specific link at the end of the survey. However, the URL that you need to set up will vary by account and study (please see Step #7 to see how to access that URL). To redirect the ending of your Qualtrics survey, add an End of Survey block at the end of your survey flow. Select Customize End of Survey and check Override Survey Options. Then choose the Redirect to a URL option and enter the specific URL in the field (e.g., see below for how it’ll look like but ignore the specific URL seen here).
  4. Next, after making these adjustments, you will want to retrieve the Prolific-specific study link to post on Prolific. To do so, send your study URL (i.e., Qualtrics link) to our lab manager Daryl, for him to create the study, and he will send you the Discoveries Online URL that includes video-recording. Then, copy the Discoveries Online URL to a text editor. You will notice that the URL ends with “?exuid=”. Add “{{%PROLIFIC_PID%}}” to the end of the URL, so the end now becomes “?exuid={{%PROLIFIC_PID%}}” (e.g., http://www.discoveriesinaction.org/accounts/studies/study_instructions_public/43/?exuid={{%PROLIFIC_PID%}}).

    After making these changes to your Qualtrics survey and study URL, you will be ready to start posting your study on Prolific (https://app.prolific.co/):
  5. You will need to register for a researcher account (i.e., select “I want to recruit participants” when registering), if you don’t have one already.
  6. Select “New study” on the left column. The first section is “STUDY DETAILS” and is mostly just your study name and description. If you select the advanced options, you will also be able to give the study a separate name for internal use. For study name, we usually use a short, engaging sentence plus a note that the study is for children between so and so years old (e.g., “Learning about People from Stories! A study for children 5-8 years old). For study description, this is an example template based on what our lab usually puts down for the description:

    This is a study produced by the [XX Lab]. We are interested in [XX]. Please have your child (ages [XX-XX]) with you to participate in this study! Note: This survey is only compatible with Chrome and Firefox browsers due to the video recording feature.
  7. (*See also Step #3*) The next section to set up is “STUDY LINK”. Enter the study link that you had prepared in Step #4 to the first field. Below it, you will see the specific URL that you will need for Step #3 to put in Qualtrics (make sure to publish your survey again after making this change). Next, check the box to confirm that you have set up the study to redirect at the end. The include special parameters option in the advanced options section should also be automatically checked when you enter the URL. If you click on Configure, this is what you should see (do not make any changes):
  8. The next section is “AUDIENCE”. You can choose for a representative study, to apply a custom pre-screening, or for everyone to see your study, depending on your study’s needs. For our lab’s developmental studies (and likely also for your Discoveries Online study), we apply a custom pre-screening, and set the following filters:
    • Year of birth of youngest child (this would be the year of birth of the youngest age of the study’s age range)
    • Current country of residence (US, UK, Canada, Ireland, Australia, New Zealand)
    • First language (English)
    • Webcam
    • Previous studies (excluding participants of any studies that are the same as or similar to the current study being run)
    For devices, we usually only select desktop, given the video and webcam technology required.
  9. The next section, PARTICIPANTS, is mostly to specify the number of participants you will need, the time estimated for them to complete the study, and the amount of compensation you will give them. Our lab usually pays $10 for a 10-15-minute study, but of course, labs can choose their own amount of compensation as they see fit (Prolific also charges a 30% service fee). The green bar under the compensation field will tell you how “good” or how “poor” the compensation is (the minimum allowed is a $6.50 hourly pay). Once you know the amount you will need to pay, you can top up your account balance accordingly.
  10. Last, you can preview your study (or save as a draft) before posting. After posting your study, you can monitor data collection for each study. As exemplified above, the embedded ID setting will allow for a unique ID to be generated and matched between your study’s Qualtrics response data and Discoveries Online video data.

Accessing data from studies hosted on Discoveries Online

Getting started

Please make sure you have the following tabs open to begin processing your data from Discoveries Online:

  1. Navigate to discoveriesinaction.org and login to the website using your credentials.
  2. Open your Processing Report document/log
  3. Open your survey platform (Qualtrics, testable, etc.,), and login to gain access to your data.

Video download

  • Click on the PANDA logo in the top left to view your dashboard, where all studies and the number of open cases for each study are listed
  • Click on the name of the study you will be processing and do the following:
    1. Click “status” to sort the participant list, and to move all open cases to the top of the list.
      Always make sure to begin by processing the earliest participant session, to ensure that families who took the study first, are getting paid first (if you are providing compensation).
    2. Download the participant’s videos one by one and rename them informatively. We suggest the following format, as an example: subjectID_studyname_testdate.
      • Please make sure you are recording the date of testing, rather than the date you are downloading the video. The date of download will always be saved as metadata in the file itself.
      • Videos are downloaded in .webm format, and need to be converted to .mp4 files in order to be opened readily on your computer.
      • We recommend that you download and organize your videos daily, and ask that you delete your videos within 3 days of downloading them.

Video conversion

We use the free, open-source video conversion software HandBrake to convert raw .webm files to usable .mp4 or .mov files. Videos in mp4 format should then be stored to your lab’s server.

Survey platform data

In cases where a participant’s video file is empty or corrupted, it is almost impossible to accurately determine (1) whether a child and parent dyad completed the study, (2) if parent consent/child assent was provided, and (3) whether the video captured the full length of the study session. For this reason, we also consider evidence of study completion (reported by the survey platform that hosts your study) when considering the validity of the data itself. The following are instructions for accessing study completion data on Qualtrics:

  1. Navigate to the Qualtrics homepage where all study projects are listed, and click on your study survey.
  2. Click on “Data & Analysis” to view your study data collected from Qualtrics.
  3. On the top right, click “Responses in Progress” to view study sessions that have not closed yet.
  4. Go through the list of sessions in progress, and manually close sessions that have sufficient study completion. What can be considered “sufficient study completion” is subjective, but here are some general rules of thumb:
    • Parents often close the browser once they have finished all target questions but before “ending” the survey (leaving study completion at 99%).
    • Sometimes parents who are familiar with taking our unmoderated remote studies may skip some of the optional questions at the end of a study and end their session (potentially leaving study completion at 84%, for example).
  5. Once you have closed the necessary sessions, navigate to the main “Recorded Responses” page, and click on the “+” symbol to add the following columns:
    • Discoveries Online ID (which can be found in the “embedded data” tab; as long as you embedded this is the survey flow of your study as described here)
    • Progress (which can be found in the “survey metadata” tab)
  6. Record session completion percentage in the processing log.