Pre-soutenance discussion

In a few weeks, all M2 students will be turning in a pre-registration document in preparation for their pre-soutenance. There is an example of a pre-registration here and some other helpful documents can be accessed from here. We invite you to post a question or comment in the discussion in this post.

Level of detail, timing, and multi-experiment research

1) How detailed should the description of the design be? 
 The document I saw yesterday does not say much about this. Would it be enough to describe the experimental conditions without all the concrete details, for instance to say things like:

 "In Condition 1, we will pair sentences of the form X with a picture in which reading R1 is true,  but not reading R2, ask subject to rate the pair on a graded scale. In condition 2...something similar  (all this without describing the concrete pictures). We predict the rating of condition 1 to be higher than condition 2, and will test this prediction by running the statistical test Z"

[In fact, quite often, in some projects I have co-supervised, there was a pilot experiment whose goal was to decide which kind of pictures should be used (for instance).]

Reply from Alex Cristia: The rule of thumb is that critical decisions need to be made prior to data analysis. On our website there is an example of a pre-registration document to illustrate the level of detail. That experiment was based on previous work, so decisions such as how many stimuli to present could be based on that prior research. However, if one knows that one will need to "play" with these parameters, then it's a good idea to reserve some time in the first semester to run a pilot, and be able to answer on these issues at the pre-soutenance. 

2) Timing: does everything have to be preregistered *before the second semester starts*, or is it ok to preregister a design later (see rationale in 3)?

Reply from Alex Cristia: The pre-soutenance has as main goal to make sure that the student (and the project) is ready for the semester. Therefore, the student should prepare ONE full pre-registeration in January, so we can provide him/her with feedback as to how this is done. When only one large and homogeneous project is planned, that single pre-registration will suffice. Notice, however, that for experimental projects we recommend students that they create a project on the Open Science Framework. This will allow them to create new pre-registrations as needed.

3) Decision trees: It seems to me that the following type of research practice is entirely ok:

1. Preregister a design whose goal is to test hypothesis H1, saying in advance how the data will be analyzed.

2. Run the experiment. Maybe you get what was predicted, maybe not, but on top of the preregistered analysis, some suggestive patterns arise, which may reveal something interesting and relevant that you did not think of, and you conduct exploratory analyses of these patterns (making clear in the resulting mémoire that these are exploratory, post-hoc, etc)
3. As a result of this exploration, you generate a new hypothesis H2 (which may or may not be compatible with H1, but is interesting in its own right, and relevant to the broad research question)
4. You now want to test H2, so you preregister a new design, with explicit predictions (pertaining to H2 but also possibly to H1), run the second experiment and analyze the results according to what was was announced in the preregistration phase.

My worry is that if students have to preregister *everything in advance*, this kind of process will not be permitted in the cogmaster, which I find a bit problematic.

In principle, one might also consider the possibility of preregistering a  "decision tree":

Run Experiment 1, run analyses A1 on the data.

If A1 returns result X, test hypothesis H2 with Experiment 2, with analyses A2. If not, test hypothesis H2' with Experiment 2', with analyses A2', etc.

In practice, however, having a full decision tree in advance might be very hard and too constraining. It should be ok to do exploratory analyses and then to test new hypotheses with a new experiment (in some cases, just by replicating the previous experiment), as long as in each case the experiment is preregistered, together with the planned analysis, and all the process is then reported in a transparent manner in the report...

Reply from Alex Cristia: Once more, the whole point of this exercice is to help students learn the "new good practices", not to constrain their work. As you explain, it is usually the case that we don't foresee every fork in the road in our real research practice -- so we don't expect students to do so either. However, the recommended practice now is that every study proposing to use inferential statistics for hypothesis testing* has an appropriately specified research plan that avoids false positives and false negatives. So the timeline you describe, with pre-registrations at each point where one hopes to test a hypothesis, is perfect. These additional pre-registrations can then be included in the thesis as appendices.

* Or perhaps one should say "every study". In any case, it won't hurt to think in advance.