Online experimenting

Using jsPsych with the servers of the ILS Labs

Introduction

The ILS Labs provide facilities that allow ILS researchers to do experiments online. Online experimenting is a good way to avoid contact in person, important in the context of the Covid-19 pandemic. It also allows you to more easily measure larger participant groups, and non-Dutch speaker groups. An obvious downside to on-line experimenting, though, is that you have far less control over the experiment compared to an experiment running in the lab.

This how-to explains how to use the new-ish infrastructure of the lab that allows researchers to conduct their research on the internet. In the lab we use jsPsych, a JavaScript software library that provides methods to present stimuli in a web browser. Any plain statically hosted website may be used to conduct an experiment, but we cannot give detailed support for other software than jsPsych.

The ILS infrastructure for online experimenting

In an online experiment there are three kinds of people with different roles: researchers/experimenters, research participants and the staff of the ILS Labs. The researchers develop an experiment that will be hosted on an ILS Labs server. The participants will visit the website created by the researchers. They will go through the experiment until the experiment finishes. After the experiment finishes, the script of the experiment will upload the collected data. The lab staff will maintain access to the servers involved in this process, to which we turn next.

Next to the people, there are two servers involved. One to provide the websites for the experiments, called the “experiment server” and one to store the data, the “data server”. The staff of the ILS Labs approve and control the experiments requested on the data server and make sure that for each experiment there is also storage for the gathered data.

Figure 1 gives an overview of the discrete steps to take in order to get an experiment running using the ILS Labs infrastructure and boilerplate jsPsych experiments. The time in this diagram flows from the top to the bottom, one could see this in a set of rows. In each row there is some action that occurs. Sometimes the diagram also indicates some moment the experimenter needs to wait for the lab staff or participants.

flow-diagram-jspsych-experiments Figure 1: A diagram that shows the interaction between the 3 kinds of people with the servers of the ILS Labs. The simplified steps indicate how to get an experiment running.

Explanation of Figure 1

Download boilerplate experiment

In order to start an experiment we advise to download one of our boilerplate experiments that are stored on www.github.com. An overview of links to the boilerplates is here (in the section “Online experiments using jsPsych”). The boilerplates contain a basic setup of an experiment, for example lexical decision or self paced reading. The scripts are typically very brief but complete examples of an experiment, with almost full support (only the ID needs to be changed) of uploading data to the dataserver.

Adapt boilerplate

The boilerplates provide a complete example experiment, but of course the stimuli and instructions should always be modified for a specific study.

Technical pilot1

Test your experiment and check whether everything works in each condition for each list. You can do this on your own computer. If things are not working as expected, try running a local web server.

If necessary adapt the boilerplate again and pilot again. Repeat these steps until you are happy with your experiment.

Unique name for you experiment folder

In order to host your experiment on the data and experiment server you will need to come up with a short and descriptive name of your experiment folder. It is recommended to use lower case names such as ungrammatical-french-eggs without any spaces. This avoids errors such as ‘page isn’t found’ because there is a small difference between Upper and lower case of letters in the name. For the experiment server, Ungrammatical-french-eggs is not the same as ungrammatical-french-eggs.

Once you have a nice descriptive name that doesn’t give away the clue about what you are researching, you can browse to the data server and request an experiment. Make sure to read the help for vital pointers. Note that all your co-researchers need to log in on the data server once, before you can add them to your experiment.

The ILS Labs staff receives your request

Once you have your experiment working, have figured out a nice name for your experiment, and have requested an experiment on the data server, the lab staff will receive a notification about your request. They will use the unique name of your experiment for a folder on the experiment server. This folder will be the container of your experiment. You’ll have to put the index.html of your experiment in this folder with all the other files that belong specifically to your experiment.

Notification of the creation of the experiment

Once the lab staff has created the folder on the experiment server, they will go to the data server themselves and approve the experiment. When they approve the experiment, you will get an e-mail that the data server is ready and that the folder for the experiment is created on the experiment server. Until you upload your own experiment there is a default page inside your experimental folder saying your experiment is under construction.

Confirmation about the approval of the experiment

You will receive information about the status of the experiment. This means you can continue your effort by obtaining the access key of your experiment from the data server via your registered experiment. You’ll need the access key later in order to have your experiment upload the data to the experiment server. We’ll elaborate on this key in the next section.

Id of an experiment

Once the experiment is approved, it gets an id. The id is a vital key, that is needed to store the results of an experiment. When a participant finishes a session, the script of the experiment wants to upload the data to the data server. The main question – looking from the eyes of the data server – is: which/whose experiment is uploading data. For this reason, the experiment sends the access key along. This key uniquely maps to one experiment and makes sure you won’t upload data to the experiment of someone else.

The id of the experiment is only relevant when uploading the results to the data server. As long as you are piloting (pilot1) the experiment on your own PC, you don’t need it, because the results are then displayed in your browser window.

If you are using a boilerplate jsPsych experiment, you can put the access key in globals.js.

To make sure your experiment uploads data to the data server, you will also need to open your experiment on the data server – see its documentation (steps 5 and 6 under ‘General overview’).

Mounting of experiment folder and uploading your experiment

Once the experiment has been approved, the lab staff will have created your experiment folder on the web server. Once you have copied your experiment to the experiment server, the website of your experiment comes online. For details, see the instructions on the data server. But hold your horses, don’t send the link to your participants yet.

Technical pilot2

Before sending the link to your participants, you need to pilot your experiment as it is on the experiment server. If something went wrong, such as a typo in the access id, you won’t receive any data at all. Hence, it is best that you figure this out before inviting your participants. Run the experiment yourself in all conditions and make sure you have obtained the results at the data server, and that the result are correct and complete.

Behavioral pilot

It is important to be sure that your intended participants understand what to do, and don’t behave wildly differently from what you’d expect. To make sure of this, ask a friend or two to do the experiment, and to give you feedback about it. Again, also make sure that the data they generate is correct and complete.

Once you are completely done with piloting, delete the data from your experiment on the data server, so that you can start the experiment with actual participants with a clean slate. You can delete the data on the page for the experiment, at the bottom of the page, using the yellow button labeled ‘Delete all data’.

Invite participants

Get in contact with your research participants. At the moment, you will need to recruit participants yourself, through friends and family, and advertising via social media. We recommend to instruct them on how to visit your website (they need to follow the link to your website). In addition, you will want to instruct them to close all competing web browsers, tabs, programs etc. You will want to ask them to visit the website on a laptop or PC (rather than a phone or tablet). Here we see a downside of online experimenting vs. experimenting in the lab: it’s trivial in the lab to close all competing programs so that all the PC’s resources go into running the experiment and get the most accurate and precise time measurements possible, but enforcing this online is a matter of goodwill on the side of the participant.

As for obtaining informed consent, you may email your participants first with information about your experiment and ask for their consent, or let the experiment do this. The jsPsych boilerplates contain a page for obtaining informed consent.

Participants receive the link to the experiment

Once the participants have received the link to the website, they can visit the online experiment.

Participants run the experiment

Once the participants have received the link to the experiment they can participate. They should pick their favorite browser and participate by visiting the link you send them. The browser will download the experiment which is hosted at the experiment server and run the experiment.
Note that although the experiment is hosted at the experiment server, the experiment actually runs inside of the browser of the participant, on their own PC.

Uploading the data

Once the participant finishes the experiment, the boilerplate experiment scripts will use a small software library of the ILS Labs to upload the experiment to the server. So the browser of the participant uploads the data to the ILS Labs data server. If you choose to roll out your own experiment not using the boiler plates or jsPsych, you can check for details about the upload process in order to upload your own experiments, on the help page of the data server.

Analyzing your research data

Periodically, you can check whether participants are actually participating in your experiment by visiting your experiment on the data server. The data at the server is growing for each participant that finishes your experiment. You can download the data in order to analyze the data.

Conclusion

The ILS Labs have created the infrastructure using two servers, one for hosting the data and one for hosting the experiment. This infrastructure allows researchers in the ILS Labs to run experiments in a online environment instead of the lab. This is a collaboration of the servers, the participants, the researchers and the staff of the ILS Labs. This howto has hopefully clarified how the different parties and servers cooperate in order to successfully run an experiment. We hope you will have a nice experience running your experiments online.